Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Alexey Goncharuk]

Alexey, Andrey, Igniters,

So what do you think? Should we proceed with a 'hacked' version of the
message factory in 2.9 and go for the runtime message generation in later
release, or keep the code clean and fix the regression in the next
releases?
Andrey, can you take a look at my change? I think it is fairly
straightforward and does not change the semantics, just skips the factory
closures for certain messages.

Personally, I would prefer fixing the regression given that we also
introduced tracing in this release.



пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:

> Alexey,
>
> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and the
> performance of ed52559eb95 is better for about 2.5% on average on our
> environment (about the same results we got benchmarking 65c30ec6 vs
> 0606f03d). We've made 24 runs for each commit of
> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients
> 50 threads each. Yardstick results for this configuration:
> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>
> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <[hidden email]
> >:
>
>> Hi Everyone,
>>
>> I posted an instruction on how to publish the docs on
>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
>> update the docs by following the instruction. Unfortunately, I won't be
>> able to spend any time on this project any longer. You can send your pull
>> requests and questions about the documentation to Denis Magda.
>>
>> -Artem
>>
>> [1] : https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>
>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> [hidden email]> wrote:
>>
>>> Alexey,
>>>
>>> I've tried to play with message factories locally, but unfortunately, I
>>> cannot spot the difference between old and new implementation in
>>> distributed benchmarks. I pushed an implementation of MessageFactoryImpl
>>> with the old switch statement to the ignite-2.9-revert-12568 branch
>>> (discussed this with Andrey Gura, the change should be compatible with
>>> the
>>> new metrics as we still use the register() mechanics).
>>>
>>> Can you check if this change makes any difference performance-wise in
>>> your
>>> environment? If yes, we can go with runtime code generation in the long
>>> term: register classes and generate a dynamic message factory with a
>>> switch
>>> statement once all messages are registered (not in 2.9 though,
>>> obviously).
>>>
>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]>:
>>>
>>> > Hello guys,
>>> >
>>> > I've tried to optimize tracing implementation (ticket [1]), it reduced
>>> the
>>> > drop, but not completely removed it.
>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
>>> > Ivan Artiukhov, can you please benchmark the patch [2] against 2.8.1
>>> > release on your environment?
>>> > With this patch on our environment, it's about a 3% drop left, it's
>>> close
>>> > to measurement error and I think such a drop is not a showstopper.
>>> Guys,
>>> > WDYT?
>>> >
>>> > Also, I found that compatibility is broken for JDBC thin driver
>>> between 2.8
>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should be
>>> > fixed in 2.9. I've prepared the patch.
>>> > Taras Ledkov, can you please review this patch?
>>> >
>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
>>> > send problem in some circumstances). I will cherry-pick it if there is
>>> no
>>> > objection.
>>> >
>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>> > [2] https://github.com/apache/ignite/pull/8223
>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>> >
>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
>>> >
>>> > > Alexey,
>>> > >
>>> > > I propose to include [1] issue to the 2.9 release. Since this issue
>>> is
>>> > > related to the new master key change functionality which haven't been
>>> > > released yet I think it will be safe to cherry-pick commit to the
>>> > > release branch.
>>> > >
>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>> > >
>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <[hidden email]>
>>> > wrote:
>>> > > >
>>> > > > Hello, Igniters.
>>> > > >
>>> > > > Alexey, please, include one more Python thin client fix [1] into
>>> the
>>> > 2.9
>>> > > release
>>> > > > It fixes kinda major issue - "Python client returns fields in wrong
>>> > > order since the 2 row when fields_count>10"
>>> > > >
>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>> > > > [2]
>>> > >
>>> >
>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>> > > >
>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>> > [hidden email]>
>>> > > написал(а):
>>> > > > >
>>> > > > > Alexey, thanks, got it. I am not sure we can optimize anything
>>> out of
>>> > > the
>>> > > > > message factory with suppliers (at least I have no ideas right
>>> now),
>>> > so
>>> > > > > most likely the only move here is to switch back to the switch
>>> > approach
>>> > > > > somehow preserving the metrics part. Probably, inlining the
>>> Ignite
>>> > > messages
>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me
>>> explore
>>> > the
>>> > > > > code a bit.
>>> > > > >
>>> > > > > P.S. I am surprised by the impact this part makes for the
>>> > performance.
>>> > > > > Message creation is indeed on the hot path, but a single virtual
>>> call
>>> > > > > should not make that much of a difference given the amount of
>>> other
>>> > > work
>>> > > > > happening during the message processing.
>>> > > > >
>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>> [hidden email]
>>> > >:
>>> > > > >
>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
>>> > Actually,
>>> > > we
>>> > > > >> were looking for a drop using bi-sect in the range between
>>> e6a7f93
>>> > > (first
>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and 6592dfa5
>>> (last
>>> > > commit in
>>> > > > >> the 2.9 branch). And we found these two problematic commits.
>>> > > > >>
>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a drop
>>> > between
>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with reverted
>>> > > IGNITE-13060
>>> > > > >> now and performance looks the same)
>>> > > > >>
>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not related
>>> to
>>> > > drop
>>> > > > >> between 2.8.1 and 2.9, but still has some performance problem,
>>> and
>>> > we
>>> > > can
>>> > > > >> win back IGNITE-13060 drop by this ticket.
>>> > > > >>
>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave it as
>>> is?
>>> > > > >>
>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
>>> refactoring)?
>>> > > > >>
>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>> > > [hidden email]
>>> > > > >>> :
>>> > > > >>
>>> > > > >>> Alexey,
>>> > > > >>>
>>> > > > >>> While investigating, I found that IGNITE-12568 has an
>>> incorrect fix
>>> > > > >>> version and is actually present in ignite-2.8.1 branch [1], so
>>> it
>>> > > cannot be
>>> > > > >>> the source of the drop against 2.8.1.
>>> > > > >>>
>>> > > > >>> P.S. Looks like we need to enforce a more accurate work with
>>> fix
>>> > > versions
>>> > > > >>> or develop some sort of tooling to verify the fix versions.
>>> > > > >>>
>>> > > > >>> --AG
>>> > > > >>>
>>> > > > >>> [1]
>>> > > > >>>
>>> > >
>>> >
>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>> > > > >>>
>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>> > > [hidden email]
>>> > > > >>>> :
>>> > > > >>>
>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>> > [hidden email]
>>> > > >:
>>> > > > >>>>
>>> > > > >>>>> Guys,
>>> > > > >>>>>
>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and IGNITE-12568
>>> > > (reverted
>>> > > > >>>>> it
>>> > > > >>>>> locally) and got the same performance as on 2.8.1
>>> > > > >>>>>
>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot paths, to
>>> > trace
>>> > > > >>>>> these
>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
>>> > > > >>>>>
>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case
>>> block was
>>> > > > >>>>> refactored to an array of message suppliers. The message
>>> factory
>>> > > is on
>>> > > > >>>>> the
>>> > > > >>>>> hot path, which explains why this commit has an impact on
>>> total
>>> > > > >>>>> performance.
>>> > > > >>>>> I've checked JIT assembly output, done some JMH
>>> microbenchmarks,
>>> > > and
>>> > > > >>>>> found
>>> > > > >>>>> that old implementation of MessageFactory.create() about
>>> 30-35%
>>> > > faster
>>> > > > >>>>> than
>>> > > > >>>>> the new one. The reason - approach with switch/case can
>>> > effectively
>>> > > > >>>>> inline
>>> > > > >>>>> message creation code, but with an array of suppliers
>>> relatively
>>> > > heavy
>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to rewrite
>>> the
>>> > code
>>> > > > >>>>> using
>>> > > > >>>>> an abstract class for suppliers instead of an interface (to
>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"), but it
>>> gives
>>> > > back
>>> > > > >>>>> only
>>> > > > >>>>> 10% of method performance and in this case, code looks ugly
>>> > > (lambdas
>>> > > > >>>>> can't
>>> > > > >>>>> be used). Currently, I can't find any more ways to optimize
>>> the
>>> > > current
>>> > > > >>>>> approach (except return to the switch/case block). Andrey
>>> Gura,
>>> > as
>>> > > the
>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
>>> > > optimization?
>>> > > > >>>>>
>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are some
>>> metrics
>>> > > > >>>>> already
>>> > > > >>>>> created, which can't be rewritten using old message factory
>>> > > > >>>>> implementation
>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>> Alexey,
>>> > > > >>>>
>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is already
>>> released
>>> > > in
>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is only
>>> present
>>> > > in Ignite
>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new metrics
>>> > > created for
>>> > > > >>>> 2.9 that depend on the new message factory to unblock the
>>> release
>>> > > and deal
>>> > > > >>>> with the optimizations in 2.10?
>>> > > > >>>>
>>> > > > >>>
>>> > > >
>>> > >
>>> >
>>>
>>

[Alexey Plekhanov]

Alexey,

We've benchmarked the whole set of tests tonight in both persistent and
in-memory modes on the current 2.9 branch (with minor tracing optimization
merged). And on our environment, we've got 0-2.5% drop for each benchmark
compared to 2.8.1. I think it is an acceptable result.
So, I'm ok with keeping the message factory as is and fix the regression in
the next release.
Guys, WDYT?

пн, 14 сент. 2020 г. в 12:41, Alexey Goncharuk <[hidden email]>:

> Alexey, Andrey, Igniters,
>
> So what do you think? Should we proceed with a 'hacked' version of the
> message factory in 2.9 and go for the runtime message generation in later
> release, or keep the code clean and fix the regression in the next
> releases?
> Andrey, can you take a look at my change? I think it is fairly
> straightforward and does not change the semantics, just skips the factory
> closures for certain messages.
>
> Personally, I would prefer fixing the regression given that we also
> introduced tracing in this release.
>
>
>
> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:
>
>> Alexey,
>>
>> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and the
>> performance of ed52559eb95 is better for about 2.5% on average on our
>> environment (about the same results we got benchmarking 65c30ec6 vs
>> 0606f03d). We've made 24 runs for each commit of
>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients
>> 50 threads each. Yardstick results for this configuration:
>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>
>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <[hidden email]
>> >:
>>
>>> Hi Everyone,
>>>
>>> I posted an instruction on how to publish the docs on
>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
>>> update the docs by following the instruction. Unfortunately, I won't be
>>> able to spend any time on this project any longer. You can send your pull
>>> requests and questions about the documentation to Denis Magda.
>>>
>>> -Artem
>>>
>>> [1] : https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>
>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>> [hidden email]> wrote:
>>>
>>>> Alexey,
>>>>
>>>> I've tried to play with message factories locally, but unfortunately, I
>>>> cannot spot the difference between old and new implementation in
>>>> distributed benchmarks. I pushed an implementation of MessageFactoryImpl
>>>> with the old switch statement to the ignite-2.9-revert-12568 branch
>>>> (discussed this with Andrey Gura, the change should be compatible with
>>>> the
>>>> new metrics as we still use the register() mechanics).
>>>>
>>>> Can you check if this change makes any difference performance-wise in
>>>> your
>>>> environment? If yes, we can go with runtime code generation in the long
>>>> term: register classes and generate a dynamic message factory with a
>>>> switch
>>>> statement once all messages are registered (not in 2.9 though,
>>>> obviously).
>>>>
>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]>:
>>>>
>>>> > Hello guys,
>>>> >
>>>> > I've tried to optimize tracing implementation (ticket [1]), it
>>>> reduced the
>>>> > drop, but not completely removed it.
>>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
>>>> > Ivan Artiukhov, can you please benchmark the patch [2] against 2.8.1
>>>> > release on your environment?
>>>> > With this patch on our environment, it's about a 3% drop left, it's
>>>> close
>>>> > to measurement error and I think such a drop is not a showstopper.
>>>> Guys,
>>>> > WDYT?
>>>> >
>>>> > Also, I found that compatibility is broken for JDBC thin driver
>>>> between 2.8
>>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should be
>>>> > fixed in 2.9. I've prepared the patch.
>>>> > Taras Ledkov, can you please review this patch?
>>>> >
>>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
>>>> > send problem in some circumstances). I will cherry-pick it if there
>>>> is no
>>>> > objection.
>>>> >
>>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>> > [2] https://github.com/apache/ignite/pull/8223
>>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>> >
>>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
>>>> >
>>>> > > Alexey,
>>>> > >
>>>> > > I propose to include [1] issue to the 2.9 release. Since this issue
>>>> is
>>>> > > related to the new master key change functionality which haven't
>>>> been
>>>> > > released yet I think it will be safe to cherry-pick commit to the
>>>> > > release branch.
>>>> > >
>>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>> > >
>>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <[hidden email]>
>>>> > wrote:
>>>> > > >
>>>> > > > Hello, Igniters.
>>>> > > >
>>>> > > > Alexey, please, include one more Python thin client fix [1] into
>>>> the
>>>> > 2.9
>>>> > > release
>>>> > > > It fixes kinda major issue - "Python client returns fields in
>>>> wrong
>>>> > > order since the 2 row when fields_count>10"
>>>> > > >
>>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>> > > > [2]
>>>> > >
>>>> >
>>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>> > > >
>>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>> > [hidden email]>
>>>> > > написал(а):
>>>> > > > >
>>>> > > > > Alexey, thanks, got it. I am not sure we can optimize anything
>>>> out of
>>>> > > the
>>>> > > > > message factory with suppliers (at least I have no ideas right
>>>> now),
>>>> > so
>>>> > > > > most likely the only move here is to switch back to the switch
>>>> > approach
>>>> > > > > somehow preserving the metrics part. Probably, inlining the
>>>> Ignite
>>>> > > messages
>>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me
>>>> explore
>>>> > the
>>>> > > > > code a bit.
>>>> > > > >
>>>> > > > > P.S. I am surprised by the impact this part makes for the
>>>> > performance.
>>>> > > > > Message creation is indeed on the hot path, but a single
>>>> virtual call
>>>> > > > > should not make that much of a difference given the amount of
>>>> other
>>>> > > work
>>>> > > > > happening during the message processing.
>>>> > > > >
>>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>> [hidden email]
>>>> > >:
>>>> > > > >
>>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
>>>> > Actually,
>>>> > > we
>>>> > > > >> were looking for a drop using bi-sect in the range between
>>>> e6a7f93
>>>> > > (first
>>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and 6592dfa5
>>>> (last
>>>> > > commit in
>>>> > > > >> the 2.9 branch). And we found these two problematic commits.
>>>> > > > >>
>>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a drop
>>>> > between
>>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with reverted
>>>> > > IGNITE-13060
>>>> > > > >> now and performance looks the same)
>>>> > > > >>
>>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>>> related to
>>>> > > drop
>>>> > > > >> between 2.8.1 and 2.9, but still has some performance problem,
>>>> and
>>>> > we
>>>> > > can
>>>> > > > >> win back IGNITE-13060 drop by this ticket.
>>>> > > > >>
>>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave it
>>>> as is?
>>>> > > > >>
>>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
>>>> refactoring)?
>>>> > > > >>
>>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>> > > [hidden email]
>>>> > > > >>> :
>>>> > > > >>
>>>> > > > >>> Alexey,
>>>> > > > >>>
>>>> > > > >>> While investigating, I found that IGNITE-12568 has an
>>>> incorrect fix
>>>> > > > >>> version and is actually present in ignite-2.8.1 branch [1],
>>>> so it
>>>> > > cannot be
>>>> > > > >>> the source of the drop against 2.8.1.
>>>> > > > >>>
>>>> > > > >>> P.S. Looks like we need to enforce a more accurate work with
>>>> fix
>>>> > > versions
>>>> > > > >>> or develop some sort of tooling to verify the fix versions.
>>>> > > > >>>
>>>> > > > >>> --AG
>>>> > > > >>>
>>>> > > > >>> [1]
>>>> > > > >>>
>>>> > >
>>>> >
>>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>> > > > >>>
>>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>> > > [hidden email]
>>>> > > > >>>> :
>>>> > > > >>>
>>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>> > [hidden email]
>>>> > > >:
>>>> > > > >>>>
>>>> > > > >>>>> Guys,
>>>> > > > >>>>>
>>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>> IGNITE-12568
>>>> > > (reverted
>>>> > > > >>>>> it
>>>> > > > >>>>> locally) and got the same performance as on 2.8.1
>>>> > > > >>>>>
>>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot paths,
>>>> to
>>>> > trace
>>>> > > > >>>>> these
>>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
>>>> > > > >>>>>
>>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case
>>>> block was
>>>> > > > >>>>> refactored to an array of message suppliers. The message
>>>> factory
>>>> > > is on
>>>> > > > >>>>> the
>>>> > > > >>>>> hot path, which explains why this commit has an impact on
>>>> total
>>>> > > > >>>>> performance.
>>>> > > > >>>>> I've checked JIT assembly output, done some JMH
>>>> microbenchmarks,
>>>> > > and
>>>> > > > >>>>> found
>>>> > > > >>>>> that old implementation of MessageFactory.create() about
>>>> 30-35%
>>>> > > faster
>>>> > > > >>>>> than
>>>> > > > >>>>> the new one. The reason - approach with switch/case can
>>>> > effectively
>>>> > > > >>>>> inline
>>>> > > > >>>>> message creation code, but with an array of suppliers
>>>> relatively
>>>> > > heavy
>>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to rewrite
>>>> the
>>>> > code
>>>> > > > >>>>> using
>>>> > > > >>>>> an abstract class for suppliers instead of an interface (to
>>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"), but it
>>>> gives
>>>> > > back
>>>> > > > >>>>> only
>>>> > > > >>>>> 10% of method performance and in this case, code looks ugly
>>>> > > (lambdas
>>>> > > > >>>>> can't
>>>> > > > >>>>> be used). Currently, I can't find any more ways to optimize
>>>> the
>>>> > > current
>>>> > > > >>>>> approach (except return to the switch/case block). Andrey
>>>> Gura,
>>>> > as
>>>> > > the
>>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
>>>> > > optimization?
>>>> > > > >>>>>
>>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are some
>>>> metrics
>>>> > > > >>>>> already
>>>> > > > >>>>> created, which can't be rewritten using old message factory
>>>> > > > >>>>> implementation
>>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>> Alexey,
>>>> > > > >>>>
>>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is already
>>>> released
>>>> > > in
>>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is only
>>>> present
>>>> > > in Ignite
>>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new metrics
>>>> > > created for
>>>> > > > >>>> 2.9 that depend on the new message factory to unblock the
>>>> release
>>>> > > and deal
>>>> > > > >>>> with the optimizations in 2.10?
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>

[Aleksandr Shapkin]

Hi guys,

 Alex,

Currently, we are working on the Apache Ignite Kubernetes Operator that
should help with packaging, deploying, and managing the application in the
k8s world. And it turned out that current docker images are not suitable
for running in k8s, cause they won't propagate the signals to the JVM
itself. For sample, you can’t just stop a k8s pod with a deployed Apache
Ignite instance gracefully.

 I’d like to include the IGNITE-13453 [1] improvement into the release scope

 I understand, that the most stabilization work has been done already, but
there are no changes in the codebase itself, the changes only affect the
docker deployment thus could be tested separately and won’t break the
existing benchmarks/tests, etc.

[1] - https://issues.apache.org/jira/browse/IGNITE-13453

вт, 15 сент. 2020 г. в 10:47, Alex Plehanov <[hidden email]>:

> Alexey,
>
> We've benchmarked the whole set of tests tonight in both persistent and
> in-memory modes on the current 2.9 branch (with minor tracing optimization
> merged). And on our environment, we've got 0-2.5% drop for each benchmark
> compared to 2.8.1. I think it is an acceptable result.
> So, I'm ok with keeping the message factory as is and fix the regression in
> the next release.
> Guys, WDYT?
>
> пн, 14 сент. 2020 г. в 12:41, Alexey Goncharuk <[hidden email]
> >:
>
> > Alexey, Andrey, Igniters,
> >
> > So what do you think? Should we proceed with a 'hacked' version of the
> > message factory in 2.9 and go for the runtime message generation in later
> > release, or keep the code clean and fix the regression in the next
> > releases?
> > Andrey, can you take a look at my change? I think it is fairly
> > straightforward and does not change the semantics, just skips the factory
> > closures for certain messages.
> >
> > Personally, I would prefer fixing the regression given that we also
> > introduced tracing in this release.
> >
> >
> >
> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:
> >
> >> Alexey,
> >>
> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and
> the
> >> performance of ed52559eb95 is better for about 2.5% on average on our
> >> environment (about the same results we got benchmarking 65c30ec6 vs
> >> 0606f03d). We've made 24 runs for each commit of
> >> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> >> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
> clients
> >> 50 threads each. Yardstick results for this configuration:
> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>
> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> [hidden email]
> >> >:
> >>
> >>> Hi Everyone,
> >>>
> >>> I posted an instruction on how to publish the docs on
> >>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
> >>> update the docs by following the instruction. Unfortunately, I won't be
> >>> able to spend any time on this project any longer. You can send your
> pull
> >>> requests and questions about the documentation to Denis Magda.
> >>>
> >>> -Artem
> >>>
> >>> [1] :
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>
> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>> [hidden email]> wrote:
> >>>
> >>>> Alexey,
> >>>>
> >>>> I've tried to play with message factories locally, but unfortunately,
> I
> >>>> cannot spot the difference between old and new implementation in
> >>>> distributed benchmarks. I pushed an implementation of
> MessageFactoryImpl
> >>>> with the old switch statement to the ignite-2.9-revert-12568 branch
> >>>> (discussed this with Andrey Gura, the change should be compatible with
> >>>> the
> >>>> new metrics as we still use the register() mechanics).
> >>>>
> >>>> Can you check if this change makes any difference performance-wise in
> >>>> your
> >>>> environment? If yes, we can go with runtime code generation in the
> long
> >>>> term: register classes and generate a dynamic message factory with a
> >>>> switch
> >>>> statement once all messages are registered (not in 2.9 though,
> >>>> obviously).
> >>>>
> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]>:
> >>>>
> >>>> > Hello guys,
> >>>> >
> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
> >>>> reduced the
> >>>> > drop, but not completely removed it.
> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against 2.8.1
> >>>> > release on your environment?
> >>>> > With this patch on our environment, it's about a 3% drop left, it's
> >>>> close
> >>>> > to measurement error and I think such a drop is not a showstopper.
> >>>> Guys,
> >>>> > WDYT?
> >>>> >
> >>>> > Also, I found that compatibility is broken for JDBC thin driver
> >>>> between 2.8
> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should be
> >>>> > fixed in 2.9. I've prepared the patch.
> >>>> > Taras Ledkov, can you please review this patch?
> >>>> >
> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
> >>>> > send problem in some circumstances). I will cherry-pick it if there
> >>>> is no
> >>>> > objection.
> >>>> >
> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>> > [2] https://github.com/apache/ignite/pull/8223
> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>> >
> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
> >>>> >
> >>>> > > Alexey,
> >>>> > >
> >>>> > > I propose to include [1] issue to the 2.9 release. Since this
> issue
> >>>> is
> >>>> > > related to the new master key change functionality which haven't
> >>>> been
> >>>> > > released yet I think it will be safe to cherry-pick commit to the
> >>>> > > release branch.
> >>>> > >
> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>> > >
> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <[hidden email]
> >
> >>>> > wrote:
> >>>> > > >
> >>>> > > > Hello, Igniters.
> >>>> > > >
> >>>> > > > Alexey, please, include one more Python thin client fix [1] into
> >>>> the
> >>>> > 2.9
> >>>> > > release
> >>>> > > > It fixes kinda major issue - "Python client returns fields in
> >>>> wrong
> >>>> > > order since the 2 row when fields_count>10"
> >>>> > > >
> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>> > > > [2]
> >>>> > >
> >>>> >
> >>>>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>> > > >
> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>> > [hidden email]>
> >>>> > > написал(а):
> >>>> > > > >
> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize anything
> >>>> out of
> >>>> > > the
> >>>> > > > > message factory with suppliers (at least I have no ideas right
> >>>> now),
> >>>> > so
> >>>> > > > > most likely the only move here is to switch back to the switch
> >>>> > approach
> >>>> > > > > somehow preserving the metrics part. Probably, inlining the
> >>>> Ignite
> >>>> > > messages
> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me
> >>>> explore
> >>>> > the
> >>>> > > > > code a bit.
> >>>> > > > >
> >>>> > > > > P.S. I am surprised by the impact this part makes for the
> >>>> > performance.
> >>>> > > > > Message creation is indeed on the hot path, but a single
> >>>> virtual call
> >>>> > > > > should not make that much of a difference given the amount of
> >>>> other
> >>>> > > work
> >>>> > > > > happening during the message processing.
> >>>> > > > >
> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>>> [hidden email]
> >>>> > >:
> >>>> > > > >
> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
> >>>> > Actually,
> >>>> > > we
> >>>> > > > >> were looking for a drop using bi-sect in the range between
> >>>> e6a7f93
> >>>> > > (first
> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and 6592dfa5
> >>>> (last
> >>>> > > commit in
> >>>> > > > >> the 2.9 branch). And we found these two problematic commits.
> >>>> > > > >>
> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a drop
> >>>> > between
> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with reverted
> >>>> > > IGNITE-13060
> >>>> > > > >> now and performance looks the same)
> >>>> > > > >>
> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >>>> related to
> >>>> > > drop
> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
> problem,
> >>>> and
> >>>> > we
> >>>> > > can
> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
> >>>> > > > >>
> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave it
> >>>> as is?
> >>>> > > > >>
> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
> >>>> refactoring)?
> >>>> > > > >>
> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>> > > [hidden email]
> >>>> > > > >>> :
> >>>> > > > >>
> >>>> > > > >>> Alexey,
> >>>> > > > >>>
> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
> >>>> incorrect fix
> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch [1],
> >>>> so it
> >>>> > > cannot be
> >>>> > > > >>> the source of the drop against 2.8.1.
> >>>> > > > >>>
> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate work with
> >>>> fix
> >>>> > > versions
> >>>> > > > >>> or develop some sort of tooling to verify the fix versions.
> >>>> > > > >>>
> >>>> > > > >>> --AG
> >>>> > > > >>>
> >>>> > > > >>> [1]
> >>>> > > > >>>
> >>>> > >
> >>>> >
> >>>>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>> > > > >>>
> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>> > > [hidden email]
> >>>> > > > >>>> :
> >>>> > > > >>>
> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>> > [hidden email]
> >>>> > > >:
> >>>> > > > >>>>
> >>>> > > > >>>>> Guys,
> >>>> > > > >>>>>
> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>>> IGNITE-12568
> >>>> > > (reverted
> >>>> > > > >>>>> it
> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
> >>>> > > > >>>>>
> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot paths,
> >>>> to
> >>>> > trace
> >>>> > > > >>>>> these
> >>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
> >>>> > > > >>>>>
> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case
> >>>> block was
> >>>> > > > >>>>> refactored to an array of message suppliers. The message
> >>>> factory
> >>>> > > is on
> >>>> > > > >>>>> the
> >>>> > > > >>>>> hot path, which explains why this commit has an impact on
> >>>> total
> >>>> > > > >>>>> performance.
> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
> >>>> microbenchmarks,
> >>>> > > and
> >>>> > > > >>>>> found
> >>>> > > > >>>>> that old implementation of MessageFactory.create() about
> >>>> 30-35%
> >>>> > > faster
> >>>> > > > >>>>> than
> >>>> > > > >>>>> the new one. The reason - approach with switch/case can
> >>>> > effectively
> >>>> > > > >>>>> inline
> >>>> > > > >>>>> message creation code, but with an array of suppliers
> >>>> relatively
> >>>> > > heavy
> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to rewrite
> >>>> the
> >>>> > code
> >>>> > > > >>>>> using
> >>>> > > > >>>>> an abstract class for suppliers instead of an interface
> (to
> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"), but
> it
> >>>> gives
> >>>> > > back
> >>>> > > > >>>>> only
> >>>> > > > >>>>> 10% of method performance and in this case, code looks
> ugly
> >>>> > > (lambdas
> >>>> > > > >>>>> can't
> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
> optimize
> >>>> the
> >>>> > > current
> >>>> > > > >>>>> approach (except return to the switch/case block). Andrey
> >>>> Gura,
> >>>> > as
> >>>> > > the
> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
> >>>> > > optimization?
> >>>> > > > >>>>>
> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are some
> >>>> metrics
> >>>> > > > >>>>> already
> >>>> > > > >>>>> created, which can't be rewritten using old message
> factory
> >>>> > > > >>>>> implementation
> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
> >>>> > > > >>>>>
> >>>> > > > >>>>
> >>>> > > > >>>> Alexey,
> >>>> > > > >>>>
> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is already
> >>>> released
> >>>> > > in
> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is only
> >>>> present
> >>>> > > in Ignite
> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> metrics
> >>>> > > created for
> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock the
> >>>> release
> >>>> > > and deal
> >>>> > > > >>>> with the optimizations in 2.10?
> >>>> > > > >>>>
> >>>> > > > >>>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
>

--
Alex.

[agura]

> So what do you think? Should we proceed with a 'hacked' version of the message factory in 2.9 and go for the runtime message generation in later release, or keep the code clean and fix the regression in the next releases?
> Andrey, can you take a look at my change? I think it is fairly straightforward and does not change the semantics, just skips the factory closures for certain messages.

IMHO 2.5% isn't too much especially because it isn't actual for all
workloads (I didn't get any significant drops during benchmarking). So
I prefer the runtime generation in later releases.

On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
<[hidden email]> wrote:

>
> Alexey, Andrey, Igniters,
>
> So what do you think? Should we proceed with a 'hacked' version of the message factory in 2.9 and go for the runtime message generation in later release, or keep the code clean and fix the regression in the next releases?
> Andrey, can you take a look at my change? I think it is fairly straightforward and does not change the semantics, just skips the factory closures for certain messages.
>
> Personally, I would prefer fixing the regression given that we also introduced tracing in this release.
>
>
>
> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:
>>
>> Alexey,
>>
>> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and the performance of ed52559eb95 is better for about 2.5% on average on our environment (about the same results we got benchmarking 65c30ec6 vs 0606f03d). We've made 24 runs for each commit of IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients 50 threads each. Yardstick results for this configuration:
>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>
>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <[hidden email]>:
>>>
>>> Hi Everyone,
>>>
>>> I posted an instruction on how to publish the docs on ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can update the docs by following the instruction. Unfortunately, I won't be able to spend any time on this project any longer. You can send your pull requests and questions about the documentation to Denis Magda.
>>>
>>> -Artem
>>>
>>> [1] : https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>
>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <[hidden email]> wrote:
>>>>
>>>> Alexey,
>>>>
>>>> I've tried to play with message factories locally, but unfortunately, I
>>>> cannot spot the difference between old and new implementation in
>>>> distributed benchmarks. I pushed an implementation of MessageFactoryImpl
>>>> with the old switch statement to the ignite-2.9-revert-12568 branch
>>>> (discussed this with Andrey Gura, the change should be compatible with the
>>>> new metrics as we still use the register() mechanics).
>>>>
>>>> Can you check if this change makes any difference performance-wise in your
>>>> environment? If yes, we can go with runtime code generation in the long
>>>> term: register classes and generate a dynamic message factory with a switch
>>>> statement once all messages are registered (not in 2.9 though, obviously).
>>>>
>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]>:
>>>>
>>>> > Hello guys,
>>>> >
>>>> > I've tried to optimize tracing implementation (ticket [1]), it reduced the
>>>> > drop, but not completely removed it.
>>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
>>>> > Ivan Artiukhov, can you please benchmark the patch [2] against 2.8.1
>>>> > release on your environment?
>>>> > With this patch on our environment, it's about a 3% drop left, it's close
>>>> > to measurement error and I think such a drop is not a showstopper. Guys,
>>>> > WDYT?
>>>> >
>>>> > Also, I found that compatibility is broken for JDBC thin driver between 2.8
>>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should be
>>>> > fixed in 2.9. I've prepared the patch.
>>>> > Taras Ledkov, can you please review this patch?
>>>> >
>>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
>>>> > send problem in some circumstances). I will cherry-pick it if there is no
>>>> > objection.
>>>> >
>>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>> > [2] https://github.com/apache/ignite/pull/8223
>>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>> >
>>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
>>>> >
>>>> > > Alexey,
>>>> > >
>>>> > > I propose to include [1] issue to the 2.9 release. Since this issue is
>>>> > > related to the new master key change functionality which haven't been
>>>> > > released yet I think it will be safe to cherry-pick commit to the
>>>> > > release branch.
>>>> > >
>>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>> > >
>>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <[hidden email]>
>>>> > wrote:
>>>> > > >
>>>> > > > Hello, Igniters.
>>>> > > >
>>>> > > > Alexey, please, include one more Python thin client fix [1] into the
>>>> > 2.9
>>>> > > release
>>>> > > > It fixes kinda major issue - "Python client returns fields in wrong
>>>> > > order since the 2 row when fields_count>10"
>>>> > > >
>>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>> > > > [2]
>>>> > >
>>>> > https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>> > > >
>>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>> > [hidden email]>
>>>> > > написал(а):
>>>> > > > >
>>>> > > > > Alexey, thanks, got it. I am not sure we can optimize anything out of
>>>> > > the
>>>> > > > > message factory with suppliers (at least I have no ideas right now),
>>>> > so
>>>> > > > > most likely the only move here is to switch back to the switch
>>>> > approach
>>>> > > > > somehow preserving the metrics part. Probably, inlining the Ignite
>>>> > > messages
>>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me explore
>>>> > the
>>>> > > > > code a bit.
>>>> > > > >
>>>> > > > > P.S. I am surprised by the impact this part makes for the
>>>> > performance.
>>>> > > > > Message creation is indeed on the hot path, but a single virtual call
>>>> > > > > should not make that much of a difference given the amount of other
>>>> > > work
>>>> > > > > happening during the message processing.
>>>> > > > >
>>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <[hidden email]
>>>> > >:
>>>> > > > >
>>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
>>>> > Actually,
>>>> > > we
>>>> > > > >> were looking for a drop using bi-sect in the range between e6a7f93
>>>> > > (first
>>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and 6592dfa5 (last
>>>> > > commit in
>>>> > > > >> the 2.9 branch). And we found these two problematic commits.
>>>> > > > >>
>>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a drop
>>>> > between
>>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with reverted
>>>> > > IGNITE-13060
>>>> > > > >> now and performance looks the same)
>>>> > > > >>
>>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not related to
>>>> > > drop
>>>> > > > >> between 2.8.1 and 2.9, but still has some performance problem, and
>>>> > we
>>>> > > can
>>>> > > > >> win back IGNITE-13060 drop by this ticket.
>>>> > > > >>
>>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave it as is?
>>>> > > > >>
>>>> > > > >> What should we do with IGNITE-12568 (MessageFactory refactoring)?
>>>> > > > >>
>>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>> > > [hidden email]
>>>> > > > >>> :
>>>> > > > >>
>>>> > > > >>> Alexey,
>>>> > > > >>>
>>>> > > > >>> While investigating, I found that IGNITE-12568 has an incorrect fix
>>>> > > > >>> version and is actually present in ignite-2.8.1 branch [1], so it
>>>> > > cannot be
>>>> > > > >>> the source of the drop against 2.8.1.
>>>> > > > >>>
>>>> > > > >>> P.S. Looks like we need to enforce a more accurate work with fix
>>>> > > versions
>>>> > > > >>> or develop some sort of tooling to verify the fix versions.
>>>> > > > >>>
>>>> > > > >>> --AG
>>>> > > > >>>
>>>> > > > >>> [1]
>>>> > > > >>>
>>>> > >
>>>> > https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>> > > > >>>
>>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>> > > [hidden email]
>>>> > > > >>>> :
>>>> > > > >>>
>>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>> > [hidden email]
>>>> > > >:
>>>> > > > >>>>
>>>> > > > >>>>> Guys,
>>>> > > > >>>>>
>>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and IGNITE-12568
>>>> > > (reverted
>>>> > > > >>>>> it
>>>> > > > >>>>> locally) and got the same performance as on 2.8.1
>>>> > > > >>>>>
>>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot paths, to
>>>> > trace
>>>> > > > >>>>> these
>>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
>>>> > > > >>>>>
>>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case block was
>>>> > > > >>>>> refactored to an array of message suppliers. The message factory
>>>> > > is on
>>>> > > > >>>>> the
>>>> > > > >>>>> hot path, which explains why this commit has an impact on total
>>>> > > > >>>>> performance.
>>>> > > > >>>>> I've checked JIT assembly output, done some JMH microbenchmarks,
>>>> > > and
>>>> > > > >>>>> found
>>>> > > > >>>>> that old implementation of MessageFactory.create() about 30-35%
>>>> > > faster
>>>> > > > >>>>> than
>>>> > > > >>>>> the new one. The reason - approach with switch/case can
>>>> > effectively
>>>> > > > >>>>> inline
>>>> > > > >>>>> message creation code, but with an array of suppliers relatively
>>>> > > heavy
>>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to rewrite the
>>>> > code
>>>> > > > >>>>> using
>>>> > > > >>>>> an abstract class for suppliers instead of an interface (to
>>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"), but it gives
>>>> > > back
>>>> > > > >>>>> only
>>>> > > > >>>>> 10% of method performance and in this case, code looks ugly
>>>> > > (lambdas
>>>> > > > >>>>> can't
>>>> > > > >>>>> be used). Currently, I can't find any more ways to optimize the
>>>> > > current
>>>> > > > >>>>> approach (except return to the switch/case block). Andrey Gura,
>>>> > as
>>>> > > the
>>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
>>>> > > optimization?
>>>> > > > >>>>>
>>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are some metrics
>>>> > > > >>>>> already
>>>> > > > >>>>> created, which can't be rewritten using old message factory
>>>> > > > >>>>> implementation
>>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>> Alexey,
>>>> > > > >>>>
>>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is already released
>>>> > > in
>>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is only present
>>>> > > in Ignite
>>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new metrics
>>>> > > created for
>>>> > > > >>>> 2.9 that depend on the new message factory to unblock the release
>>>> > > and deal
>>>> > > > >>>> with the optimizations in 2.10?
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > >
>>>> > >
>>>> >

[Alexey Plekhanov]

Guys,

During internal testing, we've found a critical bug with discovery (cluster
falls apart if two nodes segmented sequentially). This problem is not
reproduced in 2.8.1. I think we should fix it before release. Under
investigation now. I'll let you know when we get something.

чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:

> > So what do you think? Should we proceed with a 'hacked' version of the
> message factory in 2.9 and go for the runtime message generation in later
> release, or keep the code clean and fix the regression in the next releases?
> > Andrey, can you take a look at my change? I think it is fairly
> straightforward and does not change the semantics, just skips the factory
> closures for certain messages.
>
> IMHO 2.5% isn't too much especially because it isn't actual for all
> workloads (I didn't get any significant drops during benchmarking). So
> I prefer the runtime generation in later releases.
>
> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> <[hidden email]> wrote:
> >
> > Alexey, Andrey, Igniters,
> >
> > So what do you think? Should we proceed with a 'hacked' version of the
> message factory in 2.9 and go for the runtime message generation in later
> release, or keep the code clean and fix the regression in the next releases?
> > Andrey, can you take a look at my change? I think it is fairly
> straightforward and does not change the semantics, just skips the factory
> closures for certain messages.
> >
> > Personally, I would prefer fixing the regression given that we also
> introduced tracing in this release.
> >
> >
> >
> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:
> >>
> >> Alexey,
> >>
> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and
> the performance of ed52559eb95 is better for about 2.5% on average on our
> environment (about the same results we got benchmarking 65c30ec6 vs
> 0606f03d). We've made 24 runs for each commit of
> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients
> 50 threads each. Yardstick results for this configuration:
> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>
> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> [hidden email]>:
> >>>
> >>> Hi Everyone,
> >>>
> >>> I posted an instruction on how to publish the docs on
> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
> update the docs by following the instruction. Unfortunately, I won't be
> able to spend any time on this project any longer. You can send your pull
> requests and questions about the documentation to Denis Magda.
> >>>
> >>> -Artem
> >>>
> >>> [1] :
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>
> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> [hidden email]> wrote:
> >>>>
> >>>> Alexey,
> >>>>
> >>>> I've tried to play with message factories locally, but unfortunately,
> I
> >>>> cannot spot the difference between old and new implementation in
> >>>> distributed benchmarks. I pushed an implementation of
> MessageFactoryImpl
> >>>> with the old switch statement to the ignite-2.9-revert-12568 branch
> >>>> (discussed this with Andrey Gura, the change should be compatible
> with the
> >>>> new metrics as we still use the register() mechanics).
> >>>>
> >>>> Can you check if this change makes any difference performance-wise in
> your
> >>>> environment? If yes, we can go with runtime code generation in the
> long
> >>>> term: register classes and generate a dynamic message factory with a
> switch
> >>>> statement once all messages are registered (not in 2.9 though,
> obviously).
> >>>>
> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]>:
> >>>>
> >>>> > Hello guys,
> >>>> >
> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
> reduced the
> >>>> > drop, but not completely removed it.
> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against 2.8.1
> >>>> > release on your environment?
> >>>> > With this patch on our environment, it's about a 3% drop left, it's
> close
> >>>> > to measurement error and I think such a drop is not a showstopper.
> Guys,
> >>>> > WDYT?
> >>>> >
> >>>> > Also, I found that compatibility is broken for JDBC thin driver
> between 2.8
> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should be
> >>>> > fixed in 2.9. I've prepared the patch.
> >>>> > Taras Ledkov, can you please review this patch?
> >>>> >
> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
> >>>> > send problem in some circumstances). I will cherry-pick it if there
> is no
> >>>> > objection.
> >>>> >
> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>> > [2] https://github.com/apache/ignite/pull/8223
> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>> >
> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
> >>>> >
> >>>> > > Alexey,
> >>>> > >
> >>>> > > I propose to include [1] issue to the 2.9 release. Since this
> issue is
> >>>> > > related to the new master key change functionality which haven't
> been
> >>>> > > released yet I think it will be safe to cherry-pick commit to the
> >>>> > > release branch.
> >>>> > >
> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>> > >
> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <[hidden email]
> >
> >>>> > wrote:
> >>>> > > >
> >>>> > > > Hello, Igniters.
> >>>> > > >
> >>>> > > > Alexey, please, include one more Python thin client fix [1]
> into the
> >>>> > 2.9
> >>>> > > release
> >>>> > > > It fixes kinda major issue - "Python client returns fields in
> wrong
> >>>> > > order since the 2 row when fields_count>10"
> >>>> > > >
> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>> > > > [2]
> >>>> > >
> >>>> >
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>> > > >
> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>> > [hidden email]>
> >>>> > > написал(а):
> >>>> > > > >
> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize
> anything out of
> >>>> > > the
> >>>> > > > > message factory with suppliers (at least I have no ideas
> right now),
> >>>> > so
> >>>> > > > > most likely the only move here is to switch back to the switch
> >>>> > approach
> >>>> > > > > somehow preserving the metrics part. Probably, inlining the
> Ignite
> >>>> > > messages
> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me
> explore
> >>>> > the
> >>>> > > > > code a bit.
> >>>> > > > >
> >>>> > > > > P.S. I am surprised by the impact this part makes for the
> >>>> > performance.
> >>>> > > > > Message creation is indeed on the hot path, but a single
> virtual call
> >>>> > > > > should not make that much of a difference given the amount of
> other
> >>>> > > work
> >>>> > > > > happening during the message processing.
> >>>> > > > >
> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> [hidden email]
> >>>> > >:
> >>>> > > > >
> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
> >>>> > Actually,
> >>>> > > we
> >>>> > > > >> were looking for a drop using bi-sect in the range between
> e6a7f93
> >>>> > > (first
> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and 6592dfa5
> (last
> >>>> > > commit in
> >>>> > > > >> the 2.9 branch). And we found these two problematic commits.
> >>>> > > > >>
> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a drop
> >>>> > between
> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with reverted
> >>>> > > IGNITE-13060
> >>>> > > > >> now and performance looks the same)
> >>>> > > > >>
> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> related to
> >>>> > > drop
> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
> problem, and
> >>>> > we
> >>>> > > can
> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
> >>>> > > > >>
> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave it
> as is?
> >>>> > > > >>
> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
> refactoring)?
> >>>> > > > >>
> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>> > > [hidden email]
> >>>> > > > >>> :
> >>>> > > > >>
> >>>> > > > >>> Alexey,
> >>>> > > > >>>
> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
> incorrect fix
> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch [1],
> so it
> >>>> > > cannot be
> >>>> > > > >>> the source of the drop against 2.8.1.
> >>>> > > > >>>
> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate work
> with fix
> >>>> > > versions
> >>>> > > > >>> or develop some sort of tooling to verify the fix versions.
> >>>> > > > >>>
> >>>> > > > >>> --AG
> >>>> > > > >>>
> >>>> > > > >>> [1]
> >>>> > > > >>>
> >>>> > >
> >>>> >
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>> > > > >>>
> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>> > > [hidden email]
> >>>> > > > >>>> :
> >>>> > > > >>>
> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>> > [hidden email]
> >>>> > > >:
> >>>> > > > >>>>
> >>>> > > > >>>>> Guys,
> >>>> > > > >>>>>
> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
> IGNITE-12568
> >>>> > > (reverted
> >>>> > > > >>>>> it
> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
> >>>> > > > >>>>>
> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot
> paths, to
> >>>> > trace
> >>>> > > > >>>>> these
> >>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
> >>>> > > > >>>>>
> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case
> block was
> >>>> > > > >>>>> refactored to an array of message suppliers. The message
> factory
> >>>> > > is on
> >>>> > > > >>>>> the
> >>>> > > > >>>>> hot path, which explains why this commit has an impact on
> total
> >>>> > > > >>>>> performance.
> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
> microbenchmarks,
> >>>> > > and
> >>>> > > > >>>>> found
> >>>> > > > >>>>> that old implementation of MessageFactory.create() about
> 30-35%
> >>>> > > faster
> >>>> > > > >>>>> than
> >>>> > > > >>>>> the new one. The reason - approach with switch/case can
> >>>> > effectively
> >>>> > > > >>>>> inline
> >>>> > > > >>>>> message creation code, but with an array of suppliers
> relatively
> >>>> > > heavy
> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to
> rewrite the
> >>>> > code
> >>>> > > > >>>>> using
> >>>> > > > >>>>> an abstract class for suppliers instead of an interface
> (to
> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"), but
> it gives
> >>>> > > back
> >>>> > > > >>>>> only
> >>>> > > > >>>>> 10% of method performance and in this case, code looks
> ugly
> >>>> > > (lambdas
> >>>> > > > >>>>> can't
> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
> optimize the
> >>>> > > current
> >>>> > > > >>>>> approach (except return to the switch/case block). Andrey
> Gura,
> >>>> > as
> >>>> > > the
> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
> >>>> > > optimization?
> >>>> > > > >>>>>
> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are some
> metrics
> >>>> > > > >>>>> already
> >>>> > > > >>>>> created, which can't be rewritten using old message
> factory
> >>>> > > > >>>>> implementation
> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
> >>>> > > > >>>>>
> >>>> > > > >>>>
> >>>> > > > >>>> Alexey,
> >>>> > > > >>>>
> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is already
> released
> >>>> > > in
> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is only
> present
> >>>> > > in Ignite
> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> metrics
> >>>> > > created for
> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock the
> release
> >>>> > > and deal
> >>>> > > > >>>> with the optimizations in 2.10?
> >>>> > > > >>>>
> >>>> > > > >>>
> >>>> > > >
> >>>> > >
> >>>> >
>

[Alexey Plekhanov]

Guys,

I've filled the ticket with reproducer [1] for the discovery bug. This bug
caused by [2] ticket. We discussed with Vladimir Steshin privately and
decided to revert this ticket. I will do it today (after TC bot visa) if
there are no objections.

[1]: https://issues.apache.org/jira/browse/IGNITE-13465
[2]: https://issues.apache.org/jira/browse/IGNITE-13134

пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]>:

> Guys,
>
> During internal testing, we've found a critical bug with
> discovery (cluster falls apart if two nodes segmented sequentially). This
> problem is not reproduced in 2.8.1. I think we should fix it
> before release. Under investigation now. I'll let you know when we get
> something.
>
> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>
>> > So what do you think? Should we proceed with a 'hacked' version of the
>> message factory in 2.9 and go for the runtime message generation in later
>> release, or keep the code clean and fix the regression in the next releases?
>> > Andrey, can you take a look at my change? I think it is fairly
>> straightforward and does not change the semantics, just skips the factory
>> closures for certain messages.
>>
>> IMHO 2.5% isn't too much especially because it isn't actual for all
>> workloads (I didn't get any significant drops during benchmarking). So
>> I prefer the runtime generation in later releases.
>>
>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> <[hidden email]> wrote:
>> >
>> > Alexey, Andrey, Igniters,
>> >
>> > So what do you think? Should we proceed with a 'hacked' version of the
>> message factory in 2.9 and go for the runtime message generation in later
>> release, or keep the code clean and fix the regression in the next releases?
>> > Andrey, can you take a look at my change? I think it is fairly
>> straightforward and does not change the semantics, just skips the factory
>> closures for certain messages.
>> >
>> > Personally, I would prefer fixing the regression given that we also
>> introduced tracing in this release.
>> >
>> >
>> >
>> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:
>> >>
>> >> Alexey,
>> >>
>> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and
>> the performance of ed52559eb95 is better for about 2.5% on average on our
>> environment (about the same results we got benchmarking 65c30ec6 vs
>> 0606f03d). We've made 24 runs for each commit of
>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients
>> 50 threads each. Yardstick results for this configuration:
>> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >>
>> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> [hidden email]>:
>> >>>
>> >>> Hi Everyone,
>> >>>
>> >>> I posted an instruction on how to publish the docs on
>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
>> update the docs by following the instruction. Unfortunately, I won't be
>> able to spend any time on this project any longer. You can send your pull
>> requests and questions about the documentation to Denis Magda.
>> >>>
>> >>> -Artem
>> >>>
>> >>> [1] :
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >>>
>> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> [hidden email]> wrote:
>> >>>>
>> >>>> Alexey,
>> >>>>
>> >>>> I've tried to play with message factories locally, but
>> unfortunately, I
>> >>>> cannot spot the difference between old and new implementation in
>> >>>> distributed benchmarks. I pushed an implementation of
>> MessageFactoryImpl
>> >>>> with the old switch statement to the ignite-2.9-revert-12568 branch
>> >>>> (discussed this with Andrey Gura, the change should be compatible
>> with the
>> >>>> new metrics as we still use the register() mechanics).
>> >>>>
>> >>>> Can you check if this change makes any difference performance-wise
>> in your
>> >>>> environment? If yes, we can go with runtime code generation in the
>> long
>> >>>> term: register classes and generate a dynamic message factory with a
>> switch
>> >>>> statement once all messages are registered (not in 2.9 though,
>> obviously).
>> >>>>
>> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]
>> >:
>> >>>>
>> >>>> > Hello guys,
>> >>>> >
>> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
>> reduced the
>> >>>> > drop, but not completely removed it.
>> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
>> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against
>> 2.8.1
>> >>>> > release on your environment?
>> >>>> > With this patch on our environment, it's about a 3% drop left,
>> it's close
>> >>>> > to measurement error and I think such a drop is not a showstopper.
>> Guys,
>> >>>> > WDYT?
>> >>>> >
>> >>>> > Also, I found that compatibility is broken for JDBC thin driver
>> between 2.8
>> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should be
>> >>>> > fixed in 2.9. I've prepared the patch.
>> >>>> > Taras Ledkov, can you please review this patch?
>> >>>> >
>> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
>> >>>> > send problem in some circumstances). I will cherry-pick it if
>> there is no
>> >>>> > objection.
>> >>>> >
>> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
>> >>>> > [2] https://github.com/apache/ignite/pull/8223
>> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
>> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
>> >>>> >
>> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
>> >>>> >
>> >>>> > > Alexey,
>> >>>> > >
>> >>>> > > I propose to include [1] issue to the 2.9 release. Since this
>> issue is
>> >>>> > > related to the new master key change functionality which haven't
>> been
>> >>>> > > released yet I think it will be safe to cherry-pick commit to the
>> >>>> > > release branch.
>> >>>> > >
>> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
>> >>>> > >
>> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> [hidden email]>
>> >>>> > wrote:
>> >>>> > > >
>> >>>> > > > Hello, Igniters.
>> >>>> > > >
>> >>>> > > > Alexey, please, include one more Python thin client fix [1]
>> into the
>> >>>> > 2.9
>> >>>> > > release
>> >>>> > > > It fixes kinda major issue - "Python client returns fields in
>> wrong
>> >>>> > > order since the 2 row when fields_count>10"
>> >>>> > > >
>> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
>> >>>> > > > [2]
>> >>>> > >
>> >>>> >
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >>>> > > >
>> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >>>> > [hidden email]>
>> >>>> > > написал(а):
>> >>>> > > > >
>> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize
>> anything out of
>> >>>> > > the
>> >>>> > > > > message factory with suppliers (at least I have no ideas
>> right now),
>> >>>> > so
>> >>>> > > > > most likely the only move here is to switch back to the
>> switch
>> >>>> > approach
>> >>>> > > > > somehow preserving the metrics part. Probably, inlining the
>> Ignite
>> >>>> > > messages
>> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me
>> explore
>> >>>> > the
>> >>>> > > > > code a bit.
>> >>>> > > > >
>> >>>> > > > > P.S. I am surprised by the impact this part makes for the
>> >>>> > performance.
>> >>>> > > > > Message creation is indeed on the hot path, but a single
>> virtual call
>> >>>> > > > > should not make that much of a difference given the amount
>> of other
>> >>>> > > work
>> >>>> > > > > happening during the message processing.
>> >>>> > > > >
>> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> [hidden email]
>> >>>> > >:
>> >>>> > > > >
>> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
>> >>>> > Actually,
>> >>>> > > we
>> >>>> > > > >> were looking for a drop using bi-sect in the range between
>> e6a7f93
>> >>>> > > (first
>> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and 6592dfa5
>> (last
>> >>>> > > commit in
>> >>>> > > > >> the 2.9 branch). And we found these two problematic commits.
>> >>>> > > > >>
>> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a
>> drop
>> >>>> > between
>> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> reverted
>> >>>> > > IGNITE-13060
>> >>>> > > > >> now and performance looks the same)
>> >>>> > > > >>
>> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>> related to
>> >>>> > > drop
>> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
>> problem, and
>> >>>> > we
>> >>>> > > can
>> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
>> >>>> > > > >>
>> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave
>> it as is?
>> >>>> > > > >>
>> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
>> refactoring)?
>> >>>> > > > >>
>> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >>>> > > [hidden email]
>> >>>> > > > >>> :
>> >>>> > > > >>
>> >>>> > > > >>> Alexey,
>> >>>> > > > >>>
>> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
>> incorrect fix
>> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch
>> [1], so it
>> >>>> > > cannot be
>> >>>> > > > >>> the source of the drop against 2.8.1.
>> >>>> > > > >>>
>> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate work
>> with fix
>> >>>> > > versions
>> >>>> > > > >>> or develop some sort of tooling to verify the fix versions.
>> >>>> > > > >>>
>> >>>> > > > >>> --AG
>> >>>> > > > >>>
>> >>>> > > > >>> [1]
>> >>>> > > > >>>
>> >>>> > >
>> >>>> >
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >>>> > > > >>>
>> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >>>> > > [hidden email]
>> >>>> > > > >>>> :
>> >>>> > > > >>>
>> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >>>> > [hidden email]
>> >>>> > > >:
>> >>>> > > > >>>>
>> >>>> > > > >>>>> Guys,
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> IGNITE-12568
>> >>>> > > (reverted
>> >>>> > > > >>>>> it
>> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot
>> paths, to
>> >>>> > trace
>> >>>> > > > >>>>> these
>> >>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case
>> block was
>> >>>> > > > >>>>> refactored to an array of message suppliers. The message
>> factory
>> >>>> > > is on
>> >>>> > > > >>>>> the
>> >>>> > > > >>>>> hot path, which explains why this commit has an impact
>> on total
>> >>>> > > > >>>>> performance.
>> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
>> microbenchmarks,
>> >>>> > > and
>> >>>> > > > >>>>> found
>> >>>> > > > >>>>> that old implementation of MessageFactory.create() about
>> 30-35%
>> >>>> > > faster
>> >>>> > > > >>>>> than
>> >>>> > > > >>>>> the new one. The reason - approach with switch/case can
>> >>>> > effectively
>> >>>> > > > >>>>> inline
>> >>>> > > > >>>>> message creation code, but with an array of suppliers
>> relatively
>> >>>> > > heavy
>> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to
>> rewrite the
>> >>>> > code
>> >>>> > > > >>>>> using
>> >>>> > > > >>>>> an abstract class for suppliers instead of an interface
>> (to
>> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"), but
>> it gives
>> >>>> > > back
>> >>>> > > > >>>>> only
>> >>>> > > > >>>>> 10% of method performance and in this case, code looks
>> ugly
>> >>>> > > (lambdas
>> >>>> > > > >>>>> can't
>> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
>> optimize the
>> >>>> > > current
>> >>>> > > > >>>>> approach (except return to the switch/case block).
>> Andrey Gura,
>> >>>> > as
>> >>>> > > the
>> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
>> >>>> > > optimization?
>> >>>> > > > >>>>>
>> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are
>> some metrics
>> >>>> > > > >>>>> already
>> >>>> > > > >>>>> created, which can't be rewritten using old message
>> factory
>> >>>> > > > >>>>> implementation
>> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
>> >>>> > > > >>>>>
>> >>>> > > > >>>>
>> >>>> > > > >>>> Alexey,
>> >>>> > > > >>>>
>> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is already
>> released
>> >>>> > > in
>> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is only
>> present
>> >>>> > > in Ignite
>> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>> metrics
>> >>>> > > created for
>> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock the
>> release
>> >>>> > > and deal
>> >>>> > > > >>>> with the optimizations in 2.10?
>> >>>> > > > >>>>
>> >>>> > > > >>>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>>
>

[Alexey Plekhanov]

Igniters,

I've prepared release notes for the 2.9 release [1]. Can anyone review it,
please?

[1]: https://github.com/apache/ignite/pull/8273

вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]>:

> Guys,
>
> I've filled the ticket with reproducer [1] for the discovery bug. This bug
> caused by [2] ticket. We discussed with Vladimir Steshin privately and
> decided to revert this ticket. I will do it today (after TC bot visa) if
> there are no objections.
>
> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>
> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]>:
>
>> Guys,
>>
>> During internal testing, we've found a critical bug with
>> discovery (cluster falls apart if two nodes segmented sequentially). This
>> problem is not reproduced in 2.8.1. I think we should fix it
>> before release. Under investigation now. I'll let you know when we get
>> something.
>>
>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>>
>>> > So what do you think? Should we proceed with a 'hacked' version of the
>>> message factory in 2.9 and go for the runtime message generation in later
>>> release, or keep the code clean and fix the regression in the next releases?
>>> > Andrey, can you take a look at my change? I think it is fairly
>>> straightforward and does not change the semantics, just skips the factory
>>> closures for certain messages.
>>>
>>> IMHO 2.5% isn't too much especially because it isn't actual for all
>>> workloads (I didn't get any significant drops during benchmarking). So
>>> I prefer the runtime generation in later releases.
>>>
>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>> <[hidden email]> wrote:
>>> >
>>> > Alexey, Andrey, Igniters,
>>> >
>>> > So what do you think? Should we proceed with a 'hacked' version of the
>>> message factory in 2.9 and go for the runtime message generation in later
>>> release, or keep the code clean and fix the regression in the next releases?
>>> > Andrey, can you take a look at my change? I think it is fairly
>>> straightforward and does not change the semantics, just skips the factory
>>> closures for certain messages.
>>> >
>>> > Personally, I would prefer fixing the regression given that we also
>>> introduced tracing in this release.
>>> >
>>> >
>>> >
>>> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]>:
>>> >>
>>> >> Alexey,
>>> >>
>>> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and
>>> the performance of ed52559eb95 is better for about 2.5% on average on our
>>> environment (about the same results we got benchmarking 65c30ec6 vs
>>> 0606f03d). We've made 24 runs for each commit of
>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients
>>> 50 threads each. Yardstick results for this configuration:
>>> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>> >>
>>> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>> [hidden email]>:
>>> >>>
>>> >>> Hi Everyone,
>>> >>>
>>> >>> I posted an instruction on how to publish the docs on
>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
>>> update the docs by following the instruction. Unfortunately, I won't be
>>> able to spend any time on this project any longer. You can send your pull
>>> requests and questions about the documentation to Denis Magda.
>>> >>>
>>> >>> -Artem
>>> >>>
>>> >>> [1] :
>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>> >>>
>>> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>> [hidden email]> wrote:
>>> >>>>
>>> >>>> Alexey,
>>> >>>>
>>> >>>> I've tried to play with message factories locally, but
>>> unfortunately, I
>>> >>>> cannot spot the difference between old and new implementation in
>>> >>>> distributed benchmarks. I pushed an implementation of
>>> MessageFactoryImpl
>>> >>>> with the old switch statement to the ignite-2.9-revert-12568 branch
>>> >>>> (discussed this with Andrey Gura, the change should be compatible
>>> with the
>>> >>>> new metrics as we still use the register() mechanics).
>>> >>>>
>>> >>>> Can you check if this change makes any difference performance-wise
>>> in your
>>> >>>> environment? If yes, we can go with runtime code generation in the
>>> long
>>> >>>> term: register classes and generate a dynamic message factory with
>>> a switch
>>> >>>> statement once all messages are registered (not in 2.9 though,
>>> obviously).
>>> >>>>
>>> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <[hidden email]
>>> >:
>>> >>>>
>>> >>>> > Hello guys,
>>> >>>> >
>>> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
>>> reduced the
>>> >>>> > drop, but not completely removed it.
>>> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
>>> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against
>>> 2.8.1
>>> >>>> > release on your environment?
>>> >>>> > With this patch on our environment, it's about a 3% drop left,
>>> it's close
>>> >>>> > to measurement error and I think such a drop is not a
>>> showstopper. Guys,
>>> >>>> > WDYT?
>>> >>>> >
>>> >>>> > Also, I found that compatibility is broken for JDBC thin driver
>>> between 2.8
>>> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should
>>> be
>>> >>>> > fixed in 2.9. I've prepared the patch.
>>> >>>> > Taras Ledkov, can you please review this patch?
>>> >>>> >
>>> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO message
>>> >>>> > send problem in some circumstances). I will cherry-pick it if
>>> there is no
>>> >>>> > objection.
>>> >>>> >
>>> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>> >>>> > [2] https://github.com/apache/ignite/pull/8223
>>> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>> >>>> >
>>> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]>:
>>> >>>> >
>>> >>>> > > Alexey,
>>> >>>> > >
>>> >>>> > > I propose to include [1] issue to the 2.9 release. Since this
>>> issue is
>>> >>>> > > related to the new master key change functionality which
>>> haven't been
>>> >>>> > > released yet I think it will be safe to cherry-pick commit to
>>> the
>>> >>>> > > release branch.
>>> >>>> > >
>>> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>> >>>> > >
>>> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>> [hidden email]>
>>> >>>> > wrote:
>>> >>>> > > >
>>> >>>> > > > Hello, Igniters.
>>> >>>> > > >
>>> >>>> > > > Alexey, please, include one more Python thin client fix [1]
>>> into the
>>> >>>> > 2.9
>>> >>>> > > release
>>> >>>> > > > It fixes kinda major issue - "Python client returns fields in
>>> wrong
>>> >>>> > > order since the 2 row when fields_count>10"
>>> >>>> > > >
>>> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>> >>>> > > > [2]
>>> >>>> > >
>>> >>>> >
>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>> >>>> > > >
>>> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>> >>>> > [hidden email]>
>>> >>>> > > написал(а):
>>> >>>> > > > >
>>> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize
>>> anything out of
>>> >>>> > > the
>>> >>>> > > > > message factory with suppliers (at least I have no ideas
>>> right now),
>>> >>>> > so
>>> >>>> > > > > most likely the only move here is to switch back to the
>>> switch
>>> >>>> > approach
>>> >>>> > > > > somehow preserving the metrics part. Probably, inlining the
>>> Ignite
>>> >>>> > > messages
>>> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let me
>>> explore
>>> >>>> > the
>>> >>>> > > > > code a bit.
>>> >>>> > > > >
>>> >>>> > > > > P.S. I am surprised by the impact this part makes for the
>>> >>>> > performance.
>>> >>>> > > > > Message creation is indeed on the hot path, but a single
>>> virtual call
>>> >>>> > > > > should not make that much of a difference given the amount
>>> of other
>>> >>>> > > work
>>> >>>> > > > > happening during the message processing.
>>> >>>> > > > >
>>> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>> [hidden email]
>>> >>>> > >:
>>> >>>> > > > >
>>> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark results.
>>> >>>> > Actually,
>>> >>>> > > we
>>> >>>> > > > >> were looking for a drop using bi-sect in the range between
>>> e6a7f93
>>> >>>> > > (first
>>> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and
>>> 6592dfa5 (last
>>> >>>> > > commit in
>>> >>>> > > > >> the 2.9 branch). And we found these two problematic
>>> commits.
>>> >>>> > > > >>
>>> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a
>>> drop
>>> >>>> > between
>>> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>> reverted
>>> >>>> > > IGNITE-13060
>>> >>>> > > > >> now and performance looks the same)
>>> >>>> > > > >>
>>> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>> related to
>>> >>>> > > drop
>>> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
>>> problem, and
>>> >>>> > we
>>> >>>> > > can
>>> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
>>> >>>> > > > >>
>>> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave
>>> it as is?
>>> >>>> > > > >>
>>> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
>>> refactoring)?
>>> >>>> > > > >>
>>> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>> >>>> > > [hidden email]
>>> >>>> > > > >>> :
>>> >>>> > > > >>
>>> >>>> > > > >>> Alexey,
>>> >>>> > > > >>>
>>> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
>>> incorrect fix
>>> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch
>>> [1], so it
>>> >>>> > > cannot be
>>> >>>> > > > >>> the source of the drop against 2.8.1.
>>> >>>> > > > >>>
>>> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate work
>>> with fix
>>> >>>> > > versions
>>> >>>> > > > >>> or develop some sort of tooling to verify the fix
>>> versions.
>>> >>>> > > > >>>
>>> >>>> > > > >>> --AG
>>> >>>> > > > >>>
>>> >>>> > > > >>> [1]
>>> >>>> > > > >>>
>>> >>>> > >
>>> >>>> >
>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>> >>>> > > > >>>
>>> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>> >>>> > > [hidden email]
>>> >>>> > > > >>>> :
>>> >>>> > > > >>>
>>> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>> >>>> > [hidden email]
>>> >>>> > > >:
>>> >>>> > > > >>>>
>>> >>>> > > > >>>>> Guys,
>>> >>>> > > > >>>>>
>>> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>> IGNITE-12568
>>> >>>> > > (reverted
>>> >>>> > > > >>>>> it
>>> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
>>> >>>> > > > >>>>>
>>> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>> paths, to
>>> >>>> > trace
>>> >>>> > > > >>>>> these
>>> >>>> > > > >>>>> hot paths, it's clear why we have performance drop here.
>>> >>>> > > > >>>>>
>>> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) - switch/case
>>> block was
>>> >>>> > > > >>>>> refactored to an array of message suppliers. The
>>> message factory
>>> >>>> > > is on
>>> >>>> > > > >>>>> the
>>> >>>> > > > >>>>> hot path, which explains why this commit has an impact
>>> on total
>>> >>>> > > > >>>>> performance.
>>> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
>>> microbenchmarks,
>>> >>>> > > and
>>> >>>> > > > >>>>> found
>>> >>>> > > > >>>>> that old implementation of MessageFactory.create()
>>> about 30-35%
>>> >>>> > > faster
>>> >>>> > > > >>>>> than
>>> >>>> > > > >>>>> the new one. The reason - approach with switch/case can
>>> >>>> > effectively
>>> >>>> > > > >>>>> inline
>>> >>>> > > > >>>>> message creation code, but with an array of suppliers
>>> relatively
>>> >>>> > > heavy
>>> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to
>>> rewrite the
>>> >>>> > code
>>> >>>> > > > >>>>> using
>>> >>>> > > > >>>>> an abstract class for suppliers instead of an interface
>>> (to
>>> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"),
>>> but it gives
>>> >>>> > > back
>>> >>>> > > > >>>>> only
>>> >>>> > > > >>>>> 10% of method performance and in this case, code looks
>>> ugly
>>> >>>> > > (lambdas
>>> >>>> > > > >>>>> can't
>>> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
>>> optimize the
>>> >>>> > > current
>>> >>>> > > > >>>>> approach (except return to the switch/case block).
>>> Andrey Gura,
>>> >>>> > as
>>> >>>> > > the
>>> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas about
>>> >>>> > > optimization?
>>> >>>> > > > >>>>>
>>> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are
>>> some metrics
>>> >>>> > > > >>>>> already
>>> >>>> > > > >>>>> created, which can't be rewritten using old message
>>> factory
>>> >>>> > > > >>>>> implementation
>>> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
>>> >>>> > > > >>>>>
>>> >>>> > > > >>>>
>>> >>>> > > > >>>> Alexey,
>>> >>>> > > > >>>>
>>> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is
>>> already released
>>> >>>> > > in
>>> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>> only present
>>> >>>> > > in Ignite
>>> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>> metrics
>>> >>>> > > created for
>>> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock
>>> the release
>>> >>>> > > and deal
>>> >>>> > > > >>>> with the optimizations in 2.10?
>>> >>>> > > > >>>>
>>> >>>> > > > >>>
>>> >>>> > > >
>>> >>>> > >
>>> >>>> >
>>>
>>

[Alexey Goncharuk]

Saikat, Nikolay,

We have migrated a bunch of modules to ignite-extensions recently. Given
that these modules will not be available in Ignite 2.9 anymore (will
they?), should we also plan to release the extensions?

ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]>:

> Igniters,
>
> I've prepared release notes for the 2.9 release [1]. Can anyone review it,
> please?
>
> [1]: https://github.com/apache/ignite/pull/8273
>
> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]>:
>
> > Guys,
> >
> > I've filled the ticket with reproducer [1] for the discovery bug. This
> bug
> > caused by [2] ticket. We discussed with Vladimir Steshin privately and
> > decided to revert this ticket. I will do it today (after TC bot visa) if
> > there are no objections.
> >
> > [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> > [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >
> > пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]>:
> >
> >> Guys,
> >>
> >> During internal testing, we've found a critical bug with
> >> discovery (cluster falls apart if two nodes segmented sequentially).
> This
> >> problem is not reproduced in 2.8.1. I think we should fix it
> >> before release. Under investigation now. I'll let you know when we get
> >> something.
> >>
> >> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
> >>
> >>> > So what do you think? Should we proceed with a 'hacked' version of
> the
> >>> message factory in 2.9 and go for the runtime message generation in
> later
> >>> release, or keep the code clean and fix the regression in the next
> releases?
> >>> > Andrey, can you take a look at my change? I think it is fairly
> >>> straightforward and does not change the semantics, just skips the
> factory
> >>> closures for certain messages.
> >>>
> >>> IMHO 2.5% isn't too much especially because it isn't actual for all
> >>> workloads (I didn't get any significant drops during benchmarking). So
> >>> I prefer the runtime generation in later releases.
> >>>
> >>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >>> <[hidden email]> wrote:
> >>> >
> >>> > Alexey, Andrey, Igniters,
> >>> >
> >>> > So what do you think? Should we proceed with a 'hacked' version of
> the
> >>> message factory in 2.9 and go for the runtime message generation in
> later
> >>> release, or keep the code clean and fix the regression in the next
> releases?
> >>> > Andrey, can you take a look at my change? I think it is fairly
> >>> straightforward and does not change the semantics, just skips the
> factory
> >>> closures for certain messages.
> >>> >
> >>> > Personally, I would prefer fixing the regression given that we also
> >>> introduced tracing in this release.
> >>> >
> >>> >
> >>> >
> >>> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <[hidden email]
> >:
> >>> >>
> >>> >> Alexey,
> >>> >>
> >>> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
> and
> >>> the performance of ed52559eb95 is better for about 2.5% on average on
> our
> >>> environment (about the same results we got benchmarking 65c30ec6 vs
> >>> 0606f03d). We've made 24 runs for each commit of
> >>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> >>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
> clients
> >>> 50 threads each. Yardstick results for this configuration:
> >>> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >>> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>> >>
> >>> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >>> [hidden email]>:
> >>> >>>
> >>> >>> Hi Everyone,
> >>> >>>
> >>> >>> I posted an instruction on how to publish the docs on
> >>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
> >>> update the docs by following the instruction. Unfortunately, I won't be
> >>> able to spend any time on this project any longer. You can send your
> pull
> >>> requests and questions about the documentation to Denis Magda.
> >>> >>>
> >>> >>> -Artem
> >>> >>>
> >>> >>> [1] :
> >>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>> >>>
> >>> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>> [hidden email]> wrote:
> >>> >>>>
> >>> >>>> Alexey,
> >>> >>>>
> >>> >>>> I've tried to play with message factories locally, but
> >>> unfortunately, I
> >>> >>>> cannot spot the difference between old and new implementation in
> >>> >>>> distributed benchmarks. I pushed an implementation of
> >>> MessageFactoryImpl
> >>> >>>> with the old switch statement to the ignite-2.9-revert-12568
> branch
> >>> >>>> (discussed this with Andrey Gura, the change should be compatible
> >>> with the
> >>> >>>> new metrics as we still use the register() mechanics).
> >>> >>>>
> >>> >>>> Can you check if this change makes any difference performance-wise
> >>> in your
> >>> >>>> environment? If yes, we can go with runtime code generation in the
> >>> long
> >>> >>>> term: register classes and generate a dynamic message factory with
> >>> a switch
> >>> >>>> statement once all messages are registered (not in 2.9 though,
> >>> obviously).
> >>> >>>>
> >>> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> [hidden email]
> >>> >:
> >>> >>>>
> >>> >>>> > Hello guys,
> >>> >>>> >
> >>> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
> >>> reduced the
> >>> >>>> > drop, but not completely removed it.
> >>> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
> >>> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against
> >>> 2.8.1
> >>> >>>> > release on your environment?
> >>> >>>> > With this patch on our environment, it's about a 3% drop left,
> >>> it's close
> >>> >>>> > to measurement error and I think such a drop is not a
> >>> showstopper. Guys,
> >>> >>>> > WDYT?
> >>> >>>> >
> >>> >>>> > Also, I found that compatibility is broken for JDBC thin driver
> >>> between 2.8
> >>> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and should
> >>> be
> >>> >>>> > fixed in 2.9. I've prepared the patch.
> >>> >>>> > Taras Ledkov, can you please review this patch?
> >>> >>>> >
> >>> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO
> message
> >>> >>>> > send problem in some circumstances). I will cherry-pick it if
> >>> there is no
> >>> >>>> > objection.
> >>> >>>> >
> >>> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>> >>>> > [2] https://github.com/apache/ignite/pull/8223
> >>> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>> >>>> >
> >>> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <[hidden email]
> >:
> >>> >>>> >
> >>> >>>> > > Alexey,
> >>> >>>> > >
> >>> >>>> > > I propose to include [1] issue to the 2.9 release. Since this
> >>> issue is
> >>> >>>> > > related to the new master key change functionality which
> >>> haven't been
> >>> >>>> > > released yet I think it will be safe to cherry-pick commit to
> >>> the
> >>> >>>> > > release branch.
> >>> >>>> > >
> >>> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>> >>>> > >
> >>> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >>> [hidden email]>
> >>> >>>> > wrote:
> >>> >>>> > > >
> >>> >>>> > > > Hello, Igniters.
> >>> >>>> > > >
> >>> >>>> > > > Alexey, please, include one more Python thin client fix [1]
> >>> into the
> >>> >>>> > 2.9
> >>> >>>> > > release
> >>> >>>> > > > It fixes kinda major issue - "Python client returns fields
> in
> >>> wrong
> >>> >>>> > > order since the 2 row when fields_count>10"
> >>> >>>> > > >
> >>> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>> >>>> > > > [2]
> >>> >>>> > >
> >>> >>>> >
> >>>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>> >>>> > > >
> >>> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>> >>>> > [hidden email]>
> >>> >>>> > > написал(а):
> >>> >>>> > > > >
> >>> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize
> >>> anything out of
> >>> >>>> > > the
> >>> >>>> > > > > message factory with suppliers (at least I have no ideas
> >>> right now),
> >>> >>>> > so
> >>> >>>> > > > > most likely the only move here is to switch back to the
> >>> switch
> >>> >>>> > approach
> >>> >>>> > > > > somehow preserving the metrics part. Probably, inlining
> the
> >>> Ignite
> >>> >>>> > > messages
> >>> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let
> me
> >>> explore
> >>> >>>> > the
> >>> >>>> > > > > code a bit.
> >>> >>>> > > > >
> >>> >>>> > > > > P.S. I am surprised by the impact this part makes for the
> >>> >>>> > performance.
> >>> >>>> > > > > Message creation is indeed on the hot path, but a single
> >>> virtual call
> >>> >>>> > > > > should not make that much of a difference given the amount
> >>> of other
> >>> >>>> > > work
> >>> >>>> > > > > happening during the message processing.
> >>> >>>> > > > >
> >>> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>> [hidden email]
> >>> >>>> > >:
> >>> >>>> > > > >
> >>> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark
> results.
> >>> >>>> > Actually,
> >>> >>>> > > we
> >>> >>>> > > > >> were looking for a drop using bi-sect in the range
> between
> >>> e6a7f93
> >>> >>>> > > (first
> >>> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and
> >>> 6592dfa5 (last
> >>> >>>> > > commit in
> >>> >>>> > > > >> the 2.9 branch). And we found these two problematic
> >>> commits.
> >>> >>>> > > > >>
> >>> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for a
> >>> drop
> >>> >>>> > between
> >>> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >>> reverted
> >>> >>>> > > IGNITE-13060
> >>> >>>> > > > >> now and performance looks the same)
> >>> >>>> > > > >>
> >>> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >>> related to
> >>> >>>> > > drop
> >>> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
> >>> problem, and
> >>> >>>> > we
> >>> >>>> > > can
> >>> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
> >>> >>>> > > > >>
> >>> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we leave
> >>> it as is?
> >>> >>>> > > > >>
> >>> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
> >>> refactoring)?
> >>> >>>> > > > >>
> >>> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>> >>>> > > [hidden email]
> >>> >>>> > > > >>> :
> >>> >>>> > > > >>
> >>> >>>> > > > >>> Alexey,
> >>> >>>> > > > >>>
> >>> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
> >>> incorrect fix
> >>> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch
> >>> [1], so it
> >>> >>>> > > cannot be
> >>> >>>> > > > >>> the source of the drop against 2.8.1.
> >>> >>>> > > > >>>
> >>> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate work
> >>> with fix
> >>> >>>> > > versions
> >>> >>>> > > > >>> or develop some sort of tooling to verify the fix
> >>> versions.
> >>> >>>> > > > >>>
> >>> >>>> > > > >>> --AG
> >>> >>>> > > > >>>
> >>> >>>> > > > >>> [1]
> >>> >>>> > > > >>>
> >>> >>>> > >
> >>> >>>> >
> >>>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>> >>>> > > > >>>
> >>> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>> >>>> > > [hidden email]
> >>> >>>> > > > >>>> :
> >>> >>>> > > > >>>
> >>> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>> >>>> > [hidden email]
> >>> >>>> > > >:
> >>> >>>> > > > >>>>
> >>> >>>> > > > >>>>> Guys,
> >>> >>>> > > > >>>>>
> >>> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>> IGNITE-12568
> >>> >>>> > > (reverted
> >>> >>>> > > > >>>>> it
> >>> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
> >>> >>>> > > > >>>>>
> >>> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >>> paths, to
> >>> >>>> > trace
> >>> >>>> > > > >>>>> these
> >>> >>>> > > > >>>>> hot paths, it's clear why we have performance drop
> here.
> >>> >>>> > > > >>>>>
> >>> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) -
> switch/case
> >>> block was
> >>> >>>> > > > >>>>> refactored to an array of message suppliers. The
> >>> message factory
> >>> >>>> > > is on
> >>> >>>> > > > >>>>> the
> >>> >>>> > > > >>>>> hot path, which explains why this commit has an impact
> >>> on total
> >>> >>>> > > > >>>>> performance.
> >>> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
> >>> microbenchmarks,
> >>> >>>> > > and
> >>> >>>> > > > >>>>> found
> >>> >>>> > > > >>>>> that old implementation of MessageFactory.create()
> >>> about 30-35%
> >>> >>>> > > faster
> >>> >>>> > > > >>>>> than
> >>> >>>> > > > >>>>> the new one. The reason - approach with switch/case
> can
> >>> >>>> > effectively
> >>> >>>> > > > >>>>> inline
> >>> >>>> > > > >>>>> message creation code, but with an array of suppliers
> >>> relatively
> >>> >>>> > > heavy
> >>> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to
> >>> rewrite the
> >>> >>>> > code
> >>> >>>> > > > >>>>> using
> >>> >>>> > > > >>>>> an abstract class for suppliers instead of an
> interface
> >>> (to
> >>> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"),
> >>> but it gives
> >>> >>>> > > back
> >>> >>>> > > > >>>>> only
> >>> >>>> > > > >>>>> 10% of method performance and in this case, code looks
> >>> ugly
> >>> >>>> > > (lambdas
> >>> >>>> > > > >>>>> can't
> >>> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
> >>> optimize the
> >>> >>>> > > current
> >>> >>>> > > > >>>>> approach (except return to the switch/case block).
> >>> Andrey Gura,
> >>> >>>> > as
> >>> >>>> > > the
> >>> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas
> about
> >>> >>>> > > optimization?
> >>> >>>> > > > >>>>>
> >>> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are
> >>> some metrics
> >>> >>>> > > > >>>>> already
> >>> >>>> > > > >>>>> created, which can't be rewritten using old message
> >>> factory
> >>> >>>> > > > >>>>> implementation
> >>> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
> >>> >>>> > > > >>>>>
> >>> >>>> > > > >>>>
> >>> >>>> > > > >>>> Alexey,
> >>> >>>> > > > >>>>
> >>> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is
> >>> already released
> >>> >>>> > > in
> >>> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >>> only present
> >>> >>>> > > in Ignite
> >>> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >>> metrics
> >>> >>>> > > created for
> >>> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock
> >>> the release
> >>> >>>> > > and deal
> >>> >>>> > > > >>>> with the optimizations in 2.10?
> >>> >>>> > > > >>>>
> >>> >>>> > > > >>>
> >>> >>>> > > >
> >>> >>>> > >
> >>> >>>> >
> >>>
> >>
>

[Alexey Plekhanov]

Guys,

I've tried to build release candidate using TC [1]. But:
1. I can't choose a branch in this TC task (there is no such field).
2. On the "default" branch I got an error: Failed to collect changes,
error: List remote refs failed:
org.apache.http.conn.ConnectTimeoutException: Connect to
gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
internal id=74, parent id=GitBoxAsfIgnite, description: "
https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}

Is there some problem with this TC task? What am I doing wrong?

[1] :
https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild

пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <[hidden email]>:

> Saikat, Nikolay,
>
> We have migrated a bunch of modules to ignite-extensions recently. Given
> that these modules will not be available in Ignite 2.9 anymore (will
> they?), should we also plan to release the extensions?
>
> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]>:
>
> > Igniters,
> >
> > I've prepared release notes for the 2.9 release [1]. Can anyone review
> it,
> > please?
> >
> > [1]: https://github.com/apache/ignite/pull/8273
> >
> > вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]>:
> >
> > > Guys,
> > >
> > > I've filled the ticket with reproducer [1] for the discovery bug. This
> > bug
> > > caused by [2] ticket. We discussed with Vladimir Steshin privately and
> > > decided to revert this ticket. I will do it today (after TC bot visa)
> if
> > > there are no objections.
> > >
> > > [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> > > [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> > >
> > > пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]>:
> > >
> > >> Guys,
> > >>
> > >> During internal testing, we've found a critical bug with
> > >> discovery (cluster falls apart if two nodes segmented sequentially).
> > This
> > >> problem is not reproduced in 2.8.1. I think we should fix it
> > >> before release. Under investigation now. I'll let you know when we get
> > >> something.
> > >>
> > >> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
> > >>
> > >>> > So what do you think? Should we proceed with a 'hacked' version of
> > the
> > >>> message factory in 2.9 and go for the runtime message generation in
> > later
> > >>> release, or keep the code clean and fix the regression in the next
> > releases?
> > >>> > Andrey, can you take a look at my change? I think it is fairly
> > >>> straightforward and does not change the semantics, just skips the
> > factory
> > >>> closures for certain messages.
> > >>>
> > >>> IMHO 2.5% isn't too much especially because it isn't actual for all
> > >>> workloads (I didn't get any significant drops during benchmarking).
> So
> > >>> I prefer the runtime generation in later releases.
> > >>>
> > >>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> > >>> <[hidden email]> wrote:
> > >>> >
> > >>> > Alexey, Andrey, Igniters,
> > >>> >
> > >>> > So what do you think? Should we proceed with a 'hacked' version of
> > the
> > >>> message factory in 2.9 and go for the runtime message generation in
> > later
> > >>> release, or keep the code clean and fix the regression in the next
> > releases?
> > >>> > Andrey, can you take a look at my change? I think it is fairly
> > >>> straightforward and does not change the semantics, just skips the
> > factory
> > >>> closures for certain messages.
> > >>> >
> > >>> > Personally, I would prefer fixing the regression given that we also
> > >>> introduced tracing in this release.
> > >>> >
> > >>> >
> > >>> >
> > >>> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> [hidden email]
> > >:
> > >>> >>
> > >>> >> Alexey,
> > >>> >>
> > >>> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
> > and
> > >>> the performance of ed52559eb95 is better for about 2.5% on average on
> > our
> > >>> environment (about the same results we got benchmarking 65c30ec6 vs
> > >>> 0606f03d). We've made 24 runs for each commit of
> > >>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> > >>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
> > clients
> > >>> 50 threads each. Yardstick results for this configuration:
> > >>> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> > >>> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> > >>> >>
> > >>> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> > >>> [hidden email]>:
> > >>> >>>
> > >>> >>> Hi Everyone,
> > >>> >>>
> > >>> >>> I posted an instruction on how to publish the docs on
> > >>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
> > >>> update the docs by following the instruction. Unfortunately, I won't
> be
> > >>> able to spend any time on this project any longer. You can send your
> > pull
> > >>> requests and questions about the documentation to Denis Magda.
> > >>> >>>
> > >>> >>> -Artem
> > >>> >>>
> > >>> >>> [1] :
> > >>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> > >>> >>>
> > >>> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> > >>> [hidden email]> wrote:
> > >>> >>>>
> > >>> >>>> Alexey,
> > >>> >>>>
> > >>> >>>> I've tried to play with message factories locally, but
> > >>> unfortunately, I
> > >>> >>>> cannot spot the difference between old and new implementation in
> > >>> >>>> distributed benchmarks. I pushed an implementation of
> > >>> MessageFactoryImpl
> > >>> >>>> with the old switch statement to the ignite-2.9-revert-12568
> > branch
> > >>> >>>> (discussed this with Andrey Gura, the change should be
> compatible
> > >>> with the
> > >>> >>>> new metrics as we still use the register() mechanics).
> > >>> >>>>
> > >>> >>>> Can you check if this change makes any difference
> performance-wise
> > >>> in your
> > >>> >>>> environment? If yes, we can go with runtime code generation in
> the
> > >>> long
> > >>> >>>> term: register classes and generate a dynamic message factory
> with
> > >>> a switch
> > >>> >>>> statement once all messages are registered (not in 2.9 though,
> > >>> obviously).
> > >>> >>>>
> > >>> >>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> > [hidden email]
> > >>> >:
> > >>> >>>>
> > >>> >>>> > Hello guys,
> > >>> >>>> >
> > >>> >>>> > I've tried to optimize tracing implementation (ticket [1]), it
> > >>> reduced the
> > >>> >>>> > drop, but not completely removed it.
> > >>> >>>> > Ivan Rakov, Alexander Lapin, can you please review the patch?
> > >>> >>>> > Ivan Artiukhov, can you please benchmark the patch [2] against
> > >>> 2.8.1
> > >>> >>>> > release on your environment?
> > >>> >>>> > With this patch on our environment, it's about a 3% drop left,
> > >>> it's close
> > >>> >>>> > to measurement error and I think such a drop is not a
> > >>> showstopper. Guys,
> > >>> >>>> > WDYT?
> > >>> >>>> >
> > >>> >>>> > Also, I found that compatibility is broken for JDBC thin
> driver
> > >>> between 2.8
> > >>> >>>> > and 2.9 versions (ticket [3]). I think it's a blocker and
> should
> > >>> be
> > >>> >>>> > fixed in 2.9. I've prepared the patch.
> > >>> >>>> > Taras Ledkov, can you please review this patch?
> > >>> >>>> >
> > >>> >>>> > And one more ticket I propose to include into 2.9 [4] (NIO
> > message
> > >>> >>>> > send problem in some circumstances). I will cherry-pick it if
> > >>> there is no
> > >>> >>>> > objection.
> > >>> >>>> >
> > >>> >>>> > [1] https://issues.apache.org/jira/browse/IGNITE-13411
> > >>> >>>> > [2] https://github.com/apache/ignite/pull/8223
> > >>> >>>> > [3] https://issues.apache.org/jira/browse/IGNITE-13414
> > >>> >>>> > [4] https://issues.apache.org/jira/browse/IGNITE-13361
> > >>> >>>> >
> > >>> >>>> > пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> [hidden email]
> > >:
> > >>> >>>> >
> > >>> >>>> > > Alexey,
> > >>> >>>> > >
> > >>> >>>> > > I propose to include [1] issue to the 2.9 release. Since
> this
> > >>> issue is
> > >>> >>>> > > related to the new master key change functionality which
> > >>> haven't been
> > >>> >>>> > > released yet I think it will be safe to cherry-pick commit
> to
> > >>> the
> > >>> >>>> > > release branch.
> > >>> >>>> > >
> > >>> >>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-13390
> > >>> >>>> > >
> > >>> >>>> > > On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> > >>> [hidden email]>
> > >>> >>>> > wrote:
> > >>> >>>> > > >
> > >>> >>>> > > > Hello, Igniters.
> > >>> >>>> > > >
> > >>> >>>> > > > Alexey, please, include one more Python thin client fix
> [1]
> > >>> into the
> > >>> >>>> > 2.9
> > >>> >>>> > > release
> > >>> >>>> > > > It fixes kinda major issue - "Python client returns fields
> > in
> > >>> wrong
> > >>> >>>> > > order since the 2 row when fields_count>10"
> > >>> >>>> > > >
> > >>> >>>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12809
> > >>> >>>> > > > [2]
> > >>> >>>> > >
> > >>> >>>> >
> > >>>
> >
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> > >>> >>>> > > >
> > >>> >>>> > > > > 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> > >>> >>>> > [hidden email]>
> > >>> >>>> > > написал(а):
> > >>> >>>> > > > >
> > >>> >>>> > > > > Alexey, thanks, got it. I am not sure we can optimize
> > >>> anything out of
> > >>> >>>> > > the
> > >>> >>>> > > > > message factory with suppliers (at least I have no ideas
> > >>> right now),
> > >>> >>>> > so
> > >>> >>>> > > > > most likely the only move here is to switch back to the
> > >>> switch
> > >>> >>>> > approach
> > >>> >>>> > > > > somehow preserving the metrics part. Probably, inlining
> > the
> > >>> Ignite
> > >>> >>>> > > messages
> > >>> >>>> > > > > to the IgniteMessageFactoryImpl should do the trick. Let
> > me
> > >>> explore
> > >>> >>>> > the
> > >>> >>>> > > > > code a bit.
> > >>> >>>> > > > >
> > >>> >>>> > > > > P.S. I am surprised by the impact this part makes for
> the
> > >>> >>>> > performance.
> > >>> >>>> > > > > Message creation is indeed on the hot path, but a single
> > >>> virtual call
> > >>> >>>> > > > > should not make that much of a difference given the
> amount
> > >>> of other
> > >>> >>>> > > work
> > >>> >>>> > > > > happening during the message processing.
> > >>> >>>> > > > >
> > >>> >>>> > > > > пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> > >>> [hidden email]
> > >>> >>>> > >:
> > >>> >>>> > > > >
> > >>> >>>> > > > >> Alexey, sorry, I wrongly interpreted our benchmark
> > results.
> > >>> >>>> > Actually,
> > >>> >>>> > > we
> > >>> >>>> > > > >> were looking for a drop using bi-sect in the range
> > between
> > >>> e6a7f93
> > >>> >>>> > > (first
> > >>> >>>> > > > >> commit in the 2.9 branch after 2.8 branch cut) and
> > >>> 6592dfa5 (last
> > >>> >>>> > > commit in
> > >>> >>>> > > > >> the 2.9 branch). And we found these two problematic
> > >>> commits.
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> Perhaps only IGNITE-13060 (Tracing) is responsible for
> a
> > >>> drop
> > >>> >>>> > between
> > >>> >>>> > > > >> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> > >>> reverted
> > >>> >>>> > > IGNITE-13060
> > >>> >>>> > > > >> now and performance looks the same)
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> > >>> related to
> > >>> >>>> > > drop
> > >>> >>>> > > > >> between 2.8.1 and 2.9, but still has some performance
> > >>> problem, and
> > >>> >>>> > we
> > >>> >>>> > > can
> > >>> >>>> > > > >> win back IGNITE-13060 drop by this ticket.
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> Do we need more investigation on IGNITE-13060 or we
> leave
> > >>> it as is?
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> What should we do with IGNITE-12568 (MessageFactory
> > >>> refactoring)?
> > >>> >>>> > > > >>
> > >>> >>>> > > > >> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> > >>> >>>> > > [hidden email]
> > >>> >>>> > > > >>> :
> > >>> >>>> > > > >>
> > >>> >>>> > > > >>> Alexey,
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> While investigating, I found that IGNITE-12568 has an
> > >>> incorrect fix
> > >>> >>>> > > > >>> version and is actually present in ignite-2.8.1 branch
> > >>> [1], so it
> > >>> >>>> > > cannot be
> > >>> >>>> > > > >>> the source of the drop against 2.8.1.
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> P.S. Looks like we need to enforce a more accurate
> work
> > >>> with fix
> > >>> >>>> > > versions
> > >>> >>>> > > > >>> or develop some sort of tooling to verify the fix
> > >>> versions.
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> --AG
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> [1]
> > >>> >>>> > > > >>>
> > >>> >>>> > >
> > >>> >>>> >
> > >>>
> >
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> > >>> >>>> > > [hidden email]
> > >>> >>>> > > > >>>> :
> > >>> >>>> > > > >>>
> > >>> >>>> > > > >>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> > >>> >>>> > [hidden email]
> > >>> >>>> > > >:
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>>> Guys,
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> We have benchmarked 2.9 without IGNITE-13060 and
> > >>> IGNITE-12568
> > >>> >>>> > > (reverted
> > >>> >>>> > > > >>>>> it
> > >>> >>>> > > > >>>>> locally) and got the same performance as on 2.8.1
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> IGNITE-13060 (Tracing) - some code was added to hot
> > >>> paths, to
> > >>> >>>> > trace
> > >>> >>>> > > > >>>>> these
> > >>> >>>> > > > >>>>> hot paths, it's clear why we have performance drop
> > here.
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> IGNITE-12568 (MessageFactory refactoring) -
> > switch/case
> > >>> block was
> > >>> >>>> > > > >>>>> refactored to an array of message suppliers. The
> > >>> message factory
> > >>> >>>> > > is on
> > >>> >>>> > > > >>>>> the
> > >>> >>>> > > > >>>>> hot path, which explains why this commit has an
> impact
> > >>> on total
> > >>> >>>> > > > >>>>> performance.
> > >>> >>>> > > > >>>>> I've checked JIT assembly output, done some JMH
> > >>> microbenchmarks,
> > >>> >>>> > > and
> > >>> >>>> > > > >>>>> found
> > >>> >>>> > > > >>>>> that old implementation of MessageFactory.create()
> > >>> about 30-35%
> > >>> >>>> > > faster
> > >>> >>>> > > > >>>>> than
> > >>> >>>> > > > >>>>> the new one. The reason - approach with switch/case
> > can
> > >>> >>>> > effectively
> > >>> >>>> > > > >>>>> inline
> > >>> >>>> > > > >>>>> message creation code, but with an array of
> suppliers
> > >>> relatively
> > >>> >>>> > > heavy
> > >>> >>>> > > > >>>>> "invokeinterface" cannot be skipped. I've tried to
> > >>> rewrite the
> > >>> >>>> > code
> > >>> >>>> > > > >>>>> using
> > >>> >>>> > > > >>>>> an abstract class for suppliers instead of an
> > interface
> > >>> (to
> > >>> >>>> > > > >>>>> replace "invokeinterface" with the "invokevirtual"),
> > >>> but it gives
> > >>> >>>> > > back
> > >>> >>>> > > > >>>>> only
> > >>> >>>> > > > >>>>> 10% of method performance and in this case, code
> looks
> > >>> ugly
> > >>> >>>> > > (lambdas
> > >>> >>>> > > > >>>>> can't
> > >>> >>>> > > > >>>>> be used). Currently, I can't find any more ways to
> > >>> optimize the
> > >>> >>>> > > current
> > >>> >>>> > > > >>>>> approach (except return to the switch/case block).
> > >>> Andrey Gura,
> > >>> >>>> > as
> > >>> >>>> > > the
> > >>> >>>> > > > >>>>> author of IGNITE-12568, maybe you have some ideas
> > about
> > >>> >>>> > > optimization?
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>> Perhaps we should revert IGNITE-12568, but there are
> > >>> some metrics
> > >>> >>>> > > > >>>>> already
> > >>> >>>> > > > >>>>> created, which can't be rewritten using old message
> > >>> factory
> > >>> >>>> > > > >>>>> implementation
> > >>> >>>> > > > >>>>> (IGNITE-12756). Guys, WDYT?
> > >>> >>>> > > > >>>>>
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>> Alexey,
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>> I see that IGNITE-12756 (metrics improvements) is
> > >>> already released
> > >>> >>>> > > in
> > >>> >>>> > > > >>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> > >>> only present
> > >>> >>>> > > in Ignite
> > >>> >>>> > > > >>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> > >>> metrics
> > >>> >>>> > > created for
> > >>> >>>> > > > >>>> 2.9 that depend on the new message factory to unblock
> > >>> the release
> > >>> >>>> > > and deal
> > >>> >>>> > > > >>>> with the optimizations in 2.10?
> > >>> >>>> > > > >>>>
> > >>> >>>> > > > >>>
> > >>> >>>> > > >
> > >>> >>>> > >
> > >>> >>>> >
> > >>>
> > >>
> >
>

[vveider]

It seems that gitbox.apache.org is not available on CI/CD.

I will try to update VCS to GitHub.

> On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]> wrote:
>
> Guys,
>
> I've tried to build release candidate using TC [1]. But:
> 1. I can't choose a branch in this TC task (there is no such field).
> 2. On the "default" branch I got an error: Failed to collect changes,
> error: List remote refs failed:
> org.apache.http.conn.ConnectTimeoutException: Connect to
> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> internal id=74, parent id=GitBoxAsfIgnite, description: "
> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>
> Is there some problem with this TC task? What am I doing wrong?
>
> [1] :
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>
> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <[hidden email]>:
>
>> Saikat, Nikolay,
>>
>> We have migrated a bunch of modules to ignite-extensions recently. Given
>> that these modules will not be available in Ignite 2.9 anymore (will
>> they?), should we also plan to release the extensions?
>>
>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]>:
>>
>>> Igniters,
>>>
>>> I've prepared release notes for the 2.9 release [1]. Can anyone review
>> it,
>>> please?
>>>
>>> [1]: https://github.com/apache/ignite/pull/8273
>>>
>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]>:
>>>
>>>> Guys,
>>>>
>>>> I've filled the ticket with reproducer [1] for the discovery bug. This
>>> bug
>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately and
>>>> decided to revert this ticket. I will do it today (after TC bot visa)
>> if
>>>> there are no objections.
>>>>
>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>>>
>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]>:
>>>>
>>>>> Guys,
>>>>>
>>>>> During internal testing, we've found a critical bug with
>>>>> discovery (cluster falls apart if two nodes segmented sequentially).
>>> This
>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>>>> before release. Under investigation now. I'll let you know when we get
>>>>> something.
>>>>>
>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>>>>>
>>>>>>> So what do you think? Should we proceed with a 'hacked' version of
>>> the
>>>>>> message factory in 2.9 and go for the runtime message generation in
>>> later
>>>>>> release, or keep the code clean and fix the regression in the next
>>> releases?
>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>> straightforward and does not change the semantics, just skips the
>>> factory
>>>>>> closures for certain messages.
>>>>>>
>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for all
>>>>>> workloads (I didn't get any significant drops during benchmarking).
>> So
>>>>>> I prefer the runtime generation in later releases.
>>>>>>
>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>>>>> <[hidden email]> wrote:
>>>>>>>
>>>>>>> Alexey, Andrey, Igniters,
>>>>>>>
>>>>>>> So what do you think? Should we proceed with a 'hacked' version of
>>> the
>>>>>> message factory in 2.9 and go for the runtime message generation in
>>> later
>>>>>> release, or keep the code clean and fix the regression in the next
>>> releases?
>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>> straightforward and does not change the semantics, just skips the
>>> factory
>>>>>> closures for certain messages.
>>>>>>>
>>>>>>> Personally, I would prefer fixing the regression given that we also
>>>>>> introduced tracing in this release.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> [hidden email]
>>>> :
>>>>>>>>
>>>>>>>> Alexey,
>>>>>>>>
>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
>>> and
>>>>>> the performance of ed52559eb95 is better for about 2.5% on average on
>>> our
>>>>>> environment (about the same results we got benchmarking 65c30ec6 vs
>>>>>> 0606f03d). We've made 24 runs for each commit of
>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
>>> clients
>>>>>> 50 threads each. Yardstick results for this configuration:
>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>>>>>>>
>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>>>>> [hidden email]>:
>>>>>>>>>
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>> I posted an instruction on how to publish the docs on
>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
>>>>>> update the docs by following the instruction. Unfortunately, I won't
>> be
>>>>>> able to spend any time on this project any longer. You can send your
>>> pull
>>>>>> requests and questions about the documentation to Denis Magda.
>>>>>>>>>
>>>>>>>>> -Artem
>>>>>>>>>
>>>>>>>>> [1] :
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>>>>>>>
>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>>>>> [hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Alexey,
>>>>>>>>>>
>>>>>>>>>> I've tried to play with message factories locally, but
>>>>>> unfortunately, I
>>>>>>>>>> cannot spot the difference between old and new implementation in
>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>>>>> MessageFactoryImpl
>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>> branch
>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> compatible
>>>>>> with the
>>>>>>>>>> new metrics as we still use the register() mechanics).
>>>>>>>>>>
>>>>>>>>>> Can you check if this change makes any difference
>> performance-wise
>>>>>> in your
>>>>>>>>>> environment? If yes, we can go with runtime code generation in
>> the
>>>>>> long
>>>>>>>>>> term: register classes and generate a dynamic message factory
>> with
>>>>>> a switch
>>>>>>>>>> statement once all messages are registered (not in 2.9 though,
>>>>>> obviously).
>>>>>>>>>>
>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>> [hidden email]
>>>>>>> :
>>>>>>>>>>
>>>>>>>>>>> Hello guys,
>>>>>>>>>>>
>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]), it
>>>>>> reduced the
>>>>>>>>>>> drop, but not completely removed it.
>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the patch?
>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2] against
>>>>>> 2.8.1
>>>>>>>>>>> release on your environment?
>>>>>>>>>>> With this patch on our environment, it's about a 3% drop left,
>>>>>> it's close
>>>>>>>>>>> to measurement error and I think such a drop is not a
>>>>>> showstopper. Guys,
>>>>>>>>>>> WDYT?
>>>>>>>>>>>
>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> driver
>>>>>> between 2.8
>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> should
>>>>>> be
>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>>>>>>>>>>
>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>> message
>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it if
>>>>>> there is no
>>>>>>>>>>> objection.
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>>>>>>>>>
>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> [hidden email]
>>>> :
>>>>>>>>>>>
>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>
>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> this
>>>>>> issue is
>>>>>>>>>>>> related to the new master key change functionality which
>>>>>> haven't been
>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>> to
>>>>>> the
>>>>>>>>>>>> release branch.
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>>>>> [hidden email]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello, Igniters.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> [1]
>>>>>> into the
>>>>>>>>>>> 2.9
>>>>>>>>>>>> release
>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>>> in
>>>>>> wrong
>>>>>>>>>>>> order since the 2 row when fields_count>10"
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>>>>>>>>>>> [2]
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>>>>>>>>> [hidden email]>
>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>>>>> anything out of
>>>>>>>>>>>> the
>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>>>>> right now),
>>>>>>>>>>> so
>>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>>>>> switch
>>>>>>>>>>> approach
>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>> the
>>>>>> Ignite
>>>>>>>>>>>> messages
>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>> me
>>>>>> explore
>>>>>>>>>>> the
>>>>>>>>>>>>>> code a bit.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> the
>>>>>>>>>>> performance.
>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>>>>> virtual call
>>>>>>>>>>>>>> should not make that much of a difference given the
>> amount
>>>>>> of other
>>>>>>>>>>>> work
>>>>>>>>>>>>>> happening during the message processing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>>>> [hidden email]
>>>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>> results.
>>>>>>>>>>> Actually,
>>>>>>>>>>>> we
>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>> between
>>>>>> e6a7f93
>>>>>>>>>>>> (first
>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>>>>> 6592dfa5 (last
>>>>>>>>>>>> commit in
>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>>>>> commits.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>> a
>>>>>> drop
>>>>>>>>>>> between
>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>>>>> reverted
>>>>>>>>>>>> IGNITE-13060
>>>>>>>>>>>>>>> now and performance looks the same)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>>>>> related to
>>>>>>>>>>>> drop
>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>>>>> problem, and
>>>>>>>>>>> we
>>>>>>>>>>>> can
>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> leave
>>>>>> it as is?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>>>>> refactoring)?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>>>>> incorrect fix
>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>>>>> [1], so it
>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> work
>>>>>> with fix
>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>>>>> versions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --AG
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>>>> IGNITE-12568
>>>>>>>>>>>> (reverted
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>>>>> paths, to
>>>>>>>>>>> trace
>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>> here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>> switch/case
>>>>>> block was
>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>>>>> message factory
>>>>>>>>>>>> is on
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> impact
>>>>>> on total
>>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>>>>> microbenchmarks,
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>>>>> about 30-35%
>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>> can
>>>>>>>>>>> effectively
>>>>>>>>>>>>>>>>>> inline
>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> suppliers
>>>>>> relatively
>>>>>>>>>>>> heavy
>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>>>>> rewrite the
>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>> interface
>>>>>> (to
>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>>>>> but it gives
>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> looks
>>>>>> ugly
>>>>>>>>>>>> (lambdas
>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>>>>> optimize the
>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>>>>> Andrey Gura,
>>>>>>>>>>> as
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>> about
>>>>>>>>>>>> optimization?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>>>>> some metrics
>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>>>>> factory
>>>>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>>>>> already released
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>>>>> only present
>>>>>>>>>>>> in Ignite
>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>>>>> metrics
>>>>>>>>>>>> created for
>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>>>>> the release
>>>>>>>>>>>> and deal
>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>>>
>>>
>>

[Saikat Maitra]

Hi Alexey,

All the modules as planned in IEP-36
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
are migrated to ignite-extensions repository.

We can definitely plan to release ignite-extensions modules.

I have a pending PR related to the kafka module and the PR is in review
https://github.com/apache/ignite/pull/8222 but its corresponding PR has
been merged in ignite-extensions repository.

My thoughts / action items for the migration efforts and releases were
as follows:

1. Merge the changes for https://github.com/apache/ignite/pull/8222
2. Fix the upload nightly packages task for ignite-core to help fix the
travis build failure in ignite-extensions repository.
3. Review and update the README.md files for the ignite-extensions modules
as required.
4. Update the docs for ignite-extensions modules in ignite-website.
5. Release each module separately and share updates.

Please let me know if you have feedback.

Regards,
Saikat








On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]> wrote:

> It seems that gitbox.apache.org is not available on CI/CD.
>
> I will try to update VCS to GitHub.
>
>
>
>
>
> > On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]> wrote:
> >
> > Guys,
> >
> > I've tried to build release candidate using TC [1]. But:
> > 1. I can't choose a branch in this TC task (there is no such field).
> > 2. On the "default" branch I got an error: Failed to collect changes,
> > error: List remote refs failed:
> > org.apache.http.conn.ConnectTimeoutException: Connect to
> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> > internal id=74, parent id=GitBoxAsfIgnite, description: "
> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >
> > Is there some problem with this TC task? What am I doing wrong?
> >
> > [1] :
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >
> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> [hidden email]>:
> >
> >> Saikat, Nikolay,
> >>
> >> We have migrated a bunch of modules to ignite-extensions recently. Given
> >> that these modules will not be available in Ignite 2.9 anymore (will
> >> they?), should we also plan to release the extensions?
> >>
> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]>:
> >>
> >>> Igniters,
> >>>
> >>> I've prepared release notes for the 2.9 release [1]. Can anyone review
> >> it,
> >>> please?
> >>>
> >>> [1]: https://github.com/apache/ignite/pull/8273
> >>>
> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]>:
> >>>
> >>>> Guys,
> >>>>
> >>>> I've filled the ticket with reproducer [1] for the discovery bug. This
> >>> bug
> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately and
> >>>> decided to revert this ticket. I will do it today (after TC bot visa)
> >> if
> >>>> there are no objections.
> >>>>
> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >>>>
> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]
> >:
> >>>>
> >>>>> Guys,
> >>>>>
> >>>>> During internal testing, we've found a critical bug with
> >>>>> discovery (cluster falls apart if two nodes segmented sequentially).
> >>> This
> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >>>>> before release. Under investigation now. I'll let you know when we
> get
> >>>>> something.
> >>>>>
> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
> >>>>>
> >>>>>>> So what do you think? Should we proceed with a 'hacked' version of
> >>> the
> >>>>>> message factory in 2.9 and go for the runtime message generation in
> >>> later
> >>>>>> release, or keep the code clean and fix the regression in the next
> >>> releases?
> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >>>>>> straightforward and does not change the semantics, just skips the
> >>> factory
> >>>>>> closures for certain messages.
> >>>>>>
> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for all
> >>>>>> workloads (I didn't get any significant drops during benchmarking).
> >> So
> >>>>>> I prefer the runtime generation in later releases.
> >>>>>>
> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >>>>>> <[hidden email]> wrote:
> >>>>>>>
> >>>>>>> Alexey, Andrey, Igniters,
> >>>>>>>
> >>>>>>> So what do you think? Should we proceed with a 'hacked' version of
> >>> the
> >>>>>> message factory in 2.9 and go for the runtime message generation in
> >>> later
> >>>>>> release, or keep the code clean and fix the regression in the next
> >>> releases?
> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >>>>>> straightforward and does not change the semantics, just skips the
> >>> factory
> >>>>>> closures for certain messages.
> >>>>>>>
> >>>>>>> Personally, I would prefer fixing the regression given that we also
> >>>>>> introduced tracing in this release.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >> [hidden email]
> >>>> :
> >>>>>>>>
> >>>>>>>> Alexey,
> >>>>>>>>
> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
> >>> and
> >>>>>> the performance of ed52559eb95 is better for about 2.5% on average
> on
> >>> our
> >>>>>> environment (about the same results we got benchmarking 65c30ec6 vs
> >>>>>> 0606f03d). We've made 24 runs for each commit of
> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
> >>> clients
> >>>>>> 50 threads each. Yardstick results for this configuration:
> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>>>>>>>
> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >>>>>> [hidden email]>:
> >>>>>>>>>
> >>>>>>>>> Hi Everyone,
> >>>>>>>>>
> >>>>>>>>> I posted an instruction on how to publish the docs on
> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
> can
> >>>>>> update the docs by following the instruction. Unfortunately, I won't
> >> be
> >>>>>> able to spend any time on this project any longer. You can send your
> >>> pull
> >>>>>> requests and questions about the documentation to Denis Magda.
> >>>>>>>>>
> >>>>>>>>> -Artem
> >>>>>>>>>
> >>>>>>>>> [1] :
> >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>>>>>>>
> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>>>>> [hidden email]> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Alexey,
> >>>>>>>>>>
> >>>>>>>>>> I've tried to play with message factories locally, but
> >>>>>> unfortunately, I
> >>>>>>>>>> cannot spot the difference between old and new implementation in
> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >>>>>> MessageFactoryImpl
> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
> >>> branch
> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >> compatible
> >>>>>> with the
> >>>>>>>>>> new metrics as we still use the register() mechanics).
> >>>>>>>>>>
> >>>>>>>>>> Can you check if this change makes any difference
> >> performance-wise
> >>>>>> in your
> >>>>>>>>>> environment? If yes, we can go with runtime code generation in
> >> the
> >>>>>> long
> >>>>>>>>>> term: register classes and generate a dynamic message factory
> >> with
> >>>>>> a switch
> >>>>>>>>>> statement once all messages are registered (not in 2.9 though,
> >>>>>> obviously).
> >>>>>>>>>>
> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >>> [hidden email]
> >>>>>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Hello guys,
> >>>>>>>>>>>
> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]), it
> >>>>>> reduced the
> >>>>>>>>>>> drop, but not completely removed it.
> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the patch?
> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2] against
> >>>>>> 2.8.1
> >>>>>>>>>>> release on your environment?
> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop left,
> >>>>>> it's close
> >>>>>>>>>>> to measurement error and I think such a drop is not a
> >>>>>> showstopper. Guys,
> >>>>>>>>>>> WDYT?
> >>>>>>>>>>>
> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >> driver
> >>>>>> between 2.8
> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >> should
> >>>>>> be
> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >>>>>>>>>>>
> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
> >>> message
> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it if
> >>>>>> there is no
> >>>>>>>>>>> objection.
> >>>>>>>>>>>
> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>>>>>>>>>
> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >> [hidden email]
> >>>> :
> >>>>>>>>>>>
> >>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >> this
> >>>>>> issue is
> >>>>>>>>>>>> related to the new master key change functionality which
> >>>>>> haven't been
> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
> >> to
> >>>>>> the
> >>>>>>>>>>>> release branch.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >>>>>> [hidden email]>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello, Igniters.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >> [1]
> >>>>>> into the
> >>>>>>>>>>> 2.9
> >>>>>>>>>>>> release
> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
> >>> in
> >>>>>> wrong
> >>>>>>>>>>>> order since the 2 row when fields_count>10"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>>>>>>>>>>> [2]
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >>>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>>>>>>>>> [hidden email]>
> >>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >>>>>> anything out of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
> >>>>>> right now),
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
> >>>>>> switch
> >>>>>>>>>>> approach
> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
> >>> the
> >>>>>> Ignite
> >>>>>>>>>>>> messages
> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
> >>> me
> >>>>>> explore
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> code a bit.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >> the
> >>>>>>>>>>> performance.
> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
> >>>>>> virtual call
> >>>>>>>>>>>>>> should not make that much of a difference given the
> >> amount
> >>>>>> of other
> >>>>>>>>>>>> work
> >>>>>>>>>>>>>> happening during the message processing.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>>>>> [hidden email]
> >>>>>>>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >>> results.
> >>>>>>>>>>> Actually,
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >>> between
> >>>>>> e6a7f93
> >>>>>>>>>>>> (first
> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >>>>>> 6592dfa5 (last
> >>>>>>>>>>>> commit in
> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >>>>>> commits.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
> >> a
> >>>>>> drop
> >>>>>>>>>>> between
> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >>>>>> reverted
> >>>>>>>>>>>> IGNITE-13060
> >>>>>>>>>>>>>>> now and performance looks the same)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >>>>>> related to
> >>>>>>>>>>>> drop
> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >>>>>> problem, and
> >>>>>>>>>>> we
> >>>>>>>>>>>> can
> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >> leave
> >>>>>> it as is?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >>>>>> refactoring)?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
> >>>>>> incorrect fix
> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
> >>>>>> [1], so it
> >>>>>>>>>>>> cannot be
> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >> work
> >>>>>> with fix
> >>>>>>>>>>>> versions
> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >>>>>> versions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --AG
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >>>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Guys,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>>>>> IGNITE-12568
> >>>>>>>>>>>> (reverted
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >>>>>> paths, to
> >>>>>>>>>>> trace
> >>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >>> here.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >>> switch/case
> >>>>>> block was
> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >>>>>> message factory
> >>>>>>>>>>>> is on
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >> impact
> >>>>>> on total
> >>>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >>>>>> microbenchmarks,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >>>>>> about 30-35%
> >>>>>>>>>>>> faster
> >>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
> >>> can
> >>>>>>>>>>> effectively
> >>>>>>>>>>>>>>>>>> inline
> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >> suppliers
> >>>>>> relatively
> >>>>>>>>>>>> heavy
> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >>>>>> rewrite the
> >>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >>> interface
> >>>>>> (to
> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
> >>>>>> but it gives
> >>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >> looks
> >>>>>> ugly
> >>>>>>>>>>>> (lambdas
> >>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >>>>>> optimize the
> >>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >>>>>> Andrey Gura,
> >>>>>>>>>>> as
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >>> about
> >>>>>>>>>>>> optimization?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
> >>>>>> some metrics
> >>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
> >>>>>> factory
> >>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >>>>>> already released
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >>>>>> only present
> >>>>>>>>>>>> in Ignite
> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >>>>>> metrics
> >>>>>>>>>>>> created for
> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
> >>>>>> the release
> >>>>>>>>>>>> and deal
> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
>
>

[Alexey Goncharuk]

Saikat,

The plan sounds good to me! Do you have an idea for the timeline of the
module releases? Let me know if I can help in any way.

Also, I was looking into the modularization IEP and noticed that OSGI
module is discussed to be moved to the extensions, but there is no
corresponding ticket. Should I create one? I will be happy to help with
moving this module to extensions.

вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]>:

> Hi Alexey,
>
> All the modules as planned in IEP-36
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> are migrated to ignite-extensions repository.
>
> We can definitely plan to release ignite-extensions modules.
>
> I have a pending PR related to the kafka module and the PR is in review
> https://github.com/apache/ignite/pull/8222 but its corresponding PR has
> been merged in ignite-extensions repository.
>
> My thoughts / action items for the migration efforts and releases were
> as follows:
>
> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> 2. Fix the upload nightly packages task for ignite-core to help fix the
> travis build failure in ignite-extensions repository.
> 3. Review and update the README.md files for the ignite-extensions modules
> as required.
> 4. Update the docs for ignite-extensions modules in ignite-website.
> 5. Release each module separately and share updates.
>
> Please let me know if you have feedback.
>
> Regards,
> Saikat
>
>
>
>
>
>
>
>
> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]> wrote:
>
>> It seems that gitbox.apache.org is not available on CI/CD.
>>
>> I will try to update VCS to GitHub.
>>
>>
>>
>>
>>
>> > On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
>> wrote:
>> >
>> > Guys,
>> >
>> > I've tried to build release candidate using TC [1]. But:
>> > 1. I can't choose a branch in this TC task (there is no such field).
>> > 2. On the "default" branch I got an error: Failed to collect changes,
>> > error: List remote refs failed:
>> > org.apache.http.conn.ConnectTimeoutException: Connect to
>> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
>> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>> >
>> > Is there some problem with this TC task? What am I doing wrong?
>> >
>> > [1] :
>> >
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >
>> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> [hidden email]>:
>> >
>> >> Saikat, Nikolay,
>> >>
>> >> We have migrated a bunch of modules to ignite-extensions recently.
>> Given
>> >> that these modules will not be available in Ignite 2.9 anymore (will
>> >> they?), should we also plan to release the extensions?
>> >>
>> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]>:
>> >>
>> >>> Igniters,
>> >>>
>> >>> I've prepared release notes for the 2.9 release [1]. Can anyone review
>> >> it,
>> >>> please?
>> >>>
>> >>> [1]: https://github.com/apache/ignite/pull/8273
>> >>>
>> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]
>> >:
>> >>>
>> >>>> Guys,
>> >>>>
>> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>> This
>> >>> bug
>> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>> and
>> >>>> decided to revert this ticket. I will do it today (after TC bot visa)
>> >> if
>> >>>> there are no objections.
>> >>>>
>> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>> >>>>
>> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <[hidden email]
>> >:
>> >>>>
>> >>>>> Guys,
>> >>>>>
>> >>>>> During internal testing, we've found a critical bug with
>> >>>>> discovery (cluster falls apart if two nodes segmented sequentially).
>> >>> This
>> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>> >>>>> before release. Under investigation now. I'll let you know when we
>> get
>> >>>>> something.
>> >>>>>
>> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>> >>>>>
>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version of
>> >>> the
>> >>>>>> message factory in 2.9 and go for the runtime message generation in
>> >>> later
>> >>>>>> release, or keep the code clean and fix the regression in the next
>> >>> releases?
>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >>>>>> straightforward and does not change the semantics, just skips the
>> >>> factory
>> >>>>>> closures for certain messages.
>> >>>>>>
>> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for all
>> >>>>>> workloads (I didn't get any significant drops during benchmarking).
>> >> So
>> >>>>>> I prefer the runtime generation in later releases.
>> >>>>>>
>> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> >>>>>> <[hidden email]> wrote:
>> >>>>>>>
>> >>>>>>> Alexey, Andrey, Igniters,
>> >>>>>>>
>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version of
>> >>> the
>> >>>>>> message factory in 2.9 and go for the runtime message generation in
>> >>> later
>> >>>>>> release, or keep the code clean and fix the regression in the next
>> >>> releases?
>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >>>>>> straightforward and does not change the semantics, just skips the
>> >>> factory
>> >>>>>> closures for certain messages.
>> >>>>>>>
>> >>>>>>> Personally, I would prefer fixing the regression given that we
>> also
>> >>>>>> introduced tracing in this release.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> >> [hidden email]
>> >>>> :
>> >>>>>>>>
>> >>>>>>>> Alexey,
>> >>>>>>>>
>> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
>> >>> and
>> >>>>>> the performance of ed52559eb95 is better for about 2.5% on average
>> on
>> >>> our
>> >>>>>> environment (about the same results we got benchmarking 65c30ec6 vs
>> >>>>>> 0606f03d). We've made 24 runs for each commit of
>> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
>> >>> clients
>> >>>>>> 50 threads each. Yardstick results for this configuration:
>> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >>>>>>>>
>> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> >>>>>> [hidden email]>:
>> >>>>>>>>>
>> >>>>>>>>> Hi Everyone,
>> >>>>>>>>>
>> >>>>>>>>> I posted an instruction on how to publish the docs on
>> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
>> can
>> >>>>>> update the docs by following the instruction. Unfortunately, I
>> won't
>> >> be
>> >>>>>> able to spend any time on this project any longer. You can send
>> your
>> >>> pull
>> >>>>>> requests and questions about the documentation to Denis Magda.
>> >>>>>>>>>
>> >>>>>>>>> -Artem
>> >>>>>>>>>
>> >>>>>>>>> [1] :
>> >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> >>>>>> [hidden email]> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Alexey,
>> >>>>>>>>>>
>> >>>>>>>>>> I've tried to play with message factories locally, but
>> >>>>>> unfortunately, I
>> >>>>>>>>>> cannot spot the difference between old and new implementation
>> in
>> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>> >>>>>> MessageFactoryImpl
>> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>> >>> branch
>> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> >> compatible
>> >>>>>> with the
>> >>>>>>>>>> new metrics as we still use the register() mechanics).
>> >>>>>>>>>>
>> >>>>>>>>>> Can you check if this change makes any difference
>> >> performance-wise
>> >>>>>> in your
>> >>>>>>>>>> environment? If yes, we can go with runtime code generation in
>> >> the
>> >>>>>> long
>> >>>>>>>>>> term: register classes and generate a dynamic message factory
>> >> with
>> >>>>>> a switch
>> >>>>>>>>>> statement once all messages are registered (not in 2.9 though,
>> >>>>>> obviously).
>> >>>>>>>>>>
>> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>> >>> [hidden email]
>> >>>>>>> :
>> >>>>>>>>>>
>> >>>>>>>>>>> Hello guys,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]), it
>> >>>>>> reduced the
>> >>>>>>>>>>> drop, but not completely removed it.
>> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the patch?
>> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2] against
>> >>>>>> 2.8.1
>> >>>>>>>>>>> release on your environment?
>> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop left,
>> >>>>>> it's close
>> >>>>>>>>>>> to measurement error and I think such a drop is not a
>> >>>>>> showstopper. Guys,
>> >>>>>>>>>>> WDYT?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> >> driver
>> >>>>>> between 2.8
>> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> >> should
>> >>>>>> be
>> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>> >>>>>>>>>>>
>> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>> >>> message
>> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it if
>> >>>>>> there is no
>> >>>>>>>>>>> objection.
>> >>>>>>>>>>>
>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>> >>>>>>>>>>>
>> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> >> [hidden email]
>> >>>> :
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> >> this
>> >>>>>> issue is
>> >>>>>>>>>>>> related to the new master key change functionality which
>> >>>>>> haven't been
>> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>> >> to
>> >>>>>> the
>> >>>>>>>>>>>> release branch.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> >>>>>> [hidden email]>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hello, Igniters.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> >> [1]
>> >>>>>> into the
>> >>>>>>>>>>> 2.9
>> >>>>>>>>>>>> release
>> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>> >>> in
>> >>>>>> wrong
>> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>> >>>>>>>>>>>>> [2]
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>
>> >>>
>> >>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >>>>>>>>>>> [hidden email]>
>> >>>>>>>>>>>> написал(а):
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>> >>>>>> anything out of
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>> >>>>>> right now),
>> >>>>>>>>>>> so
>> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>> >>>>>> switch
>> >>>>>>>>>>> approach
>> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>> >>> the
>> >>>>>> Ignite
>> >>>>>>>>>>>> messages
>> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>> >>> me
>> >>>>>> explore
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>>> code a bit.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> >> the
>> >>>>>>>>>>> performance.
>> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>> >>>>>> virtual call
>> >>>>>>>>>>>>>> should not make that much of a difference given the
>> >> amount
>> >>>>>> of other
>> >>>>>>>>>>>> work
>> >>>>>>>>>>>>>> happening during the message processing.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> >>>>>> [hidden email]
>> >>>>>>>>>>>> :
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>> >>> results.
>> >>>>>>>>>>> Actually,
>> >>>>>>>>>>>> we
>> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>> >>> between
>> >>>>>> e6a7f93
>> >>>>>>>>>>>> (first
>> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>> >>>>>> 6592dfa5 (last
>> >>>>>>>>>>>> commit in
>> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>> >>>>>> commits.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>> >> a
>> >>>>>> drop
>> >>>>>>>>>>> between
>> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> >>>>>> reverted
>> >>>>>>>>>>>> IGNITE-13060
>> >>>>>>>>>>>>>>> now and performance looks the same)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>> >>>>>> related to
>> >>>>>>>>>>>> drop
>> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>> >>>>>> problem, and
>> >>>>>>>>>>> we
>> >>>>>>>>>>>> can
>> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> >> leave
>> >>>>>> it as is?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>> >>>>>> refactoring)?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >>>>>>>>>>>> [hidden email]
>> >>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>> >>>>>> incorrect fix
>> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>> >>>>>> [1], so it
>> >>>>>>>>>>>> cannot be
>> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> >> work
>> >>>>>> with fix
>> >>>>>>>>>>>> versions
>> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>> >>>>>> versions.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> --AG
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> [1]
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>
>> >>>
>> >>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >>>>>>>>>>>> [hidden email]
>> >>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >>>>>>>>>>> [hidden email]
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Guys,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> >>>>>> IGNITE-12568
>> >>>>>>>>>>>> (reverted
>> >>>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>> >>>>>> paths, to
>> >>>>>>>>>>> trace
>> >>>>>>>>>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>> >>> here.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>> >>> switch/case
>> >>>>>> block was
>> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>> >>>>>> message factory
>> >>>>>>>>>>>> is on
>> >>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> >> impact
>> >>>>>> on total
>> >>>>>>>>>>>>>>>>>> performance.
>> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>> >>>>>> microbenchmarks,
>> >>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>> found
>> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>> >>>>>> about 30-35%
>> >>>>>>>>>>>> faster
>> >>>>>>>>>>>>>>>>>> than
>> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>> >>> can
>> >>>>>>>>>>> effectively
>> >>>>>>>>>>>>>>>>>> inline
>> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> >> suppliers
>> >>>>>> relatively
>> >>>>>>>>>>>> heavy
>> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>> >>>>>> rewrite the
>> >>>>>>>>>>> code
>> >>>>>>>>>>>>>>>>>> using
>> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>> >>> interface
>> >>>>>> (to
>> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>> >>>>>> but it gives
>> >>>>>>>>>>>> back
>> >>>>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> >> looks
>> >>>>>> ugly
>> >>>>>>>>>>>> (lambdas
>> >>>>>>>>>>>>>>>>>> can't
>> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>> >>>>>> optimize the
>> >>>>>>>>>>>> current
>> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>> >>>>>> Andrey Gura,
>> >>>>>>>>>>> as
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>> >>> about
>> >>>>>>>>>>>> optimization?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>> >>>>>> some metrics
>> >>>>>>>>>>>>>>>>>> already
>> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>> >>>>>> factory
>> >>>>>>>>>>>>>>>>>> implementation
>> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>> >>>>>> already released
>> >>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>> >>>>>> only present
>> >>>>>>>>>>>> in Ignite
>> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>> >>>>>> metrics
>> >>>>>>>>>>>> created for
>> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>> >>>>>> the release
>> >>>>>>>>>>>> and deal
>> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> >>
>>
>>

[Saikat Maitra]

Hi Alexey,

Thank you for your email. Yes, I can certainly use help in the release
process. I am thinking around the month of Nov for the releases after
the In-Memory Computing Summit Virtual 2020 but any of the module can be
released before that timeline. I have a tech talk coming up on Oct 28 and I
may get a little occupied for the same.

https://www.imcsummit.org/2020/virtual/agenda/schedule/day-1

Also, I have not created any ticket for the OSGI module, please feel free
to create a ticket and I can help with the migration.

Regards,
Saikat

On Tue, Oct 6, 2020 at 9:19 AM Alexey Goncharuk <[hidden email]>
wrote:

> Saikat,
>
> The plan sounds good to me! Do you have an idea for the timeline of the
> module releases? Let me know if I can help in any way.
>
> Also, I was looking into the modularization IEP and noticed that OSGI
> module is discussed to be moved to the extensions, but there is no
> corresponding ticket. Should I create one? I will be happy to help with
> moving this module to extensions.
>
> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]>:
>
>> Hi Alexey,
>>
>> All the modules as planned in IEP-36
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> are migrated to ignite-extensions repository.
>>
>> We can definitely plan to release ignite-extensions modules.
>>
>> I have a pending PR related to the kafka module and the PR is in review
>> https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>> been merged in ignite-extensions repository.
>>
>> My thoughts / action items for the migration efforts and releases were
>> as follows:
>>
>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>> 2. Fix the upload nightly packages task for ignite-core to help fix the
>> travis build failure in ignite-extensions repository.
>> 3. Review and update the README.md files for the ignite-extensions
>> modules as required.
>> 4. Update the docs for ignite-extensions modules in ignite-website.
>> 5. Release each module separately and share updates.
>>
>> Please let me know if you have feedback.
>>
>> Regards,
>> Saikat
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]> wrote:
>>
>>> It seems that gitbox.apache.org is not available on CI/CD.
>>>
>>> I will try to update VCS to GitHub.
>>>
>>>
>>>
>>>
>>>
>>> > On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
>>> wrote:
>>> >
>>> > Guys,
>>> >
>>> > I've tried to build release candidate using TC [1]. But:
>>> > 1. I can't choose a branch in this TC task (there is no such field).
>>> > 2. On the "default" branch I got an error: Failed to collect changes,
>>> > error: List remote refs failed:
>>> > org.apache.http.conn.ConnectTimeoutException: Connect to
>>> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
>>> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>>> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>> >
>>> > Is there some problem with this TC task? What am I doing wrong?
>>> >
>>> > [1] :
>>> >
>>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>> >
>>> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>> [hidden email]>:
>>> >
>>> >> Saikat, Nikolay,
>>> >>
>>> >> We have migrated a bunch of modules to ignite-extensions recently.
>>> Given
>>> >> that these modules will not be available in Ignite 2.9 anymore (will
>>> >> they?), should we also plan to release the extensions?
>>> >>
>>> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]
>>> >:
>>> >>
>>> >>> Igniters,
>>> >>>
>>> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
>>> review
>>> >> it,
>>> >>> please?
>>> >>>
>>> >>> [1]: https://github.com/apache/ignite/pull/8273
>>> >>>
>>> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <[hidden email]
>>> >:
>>> >>>
>>> >>>> Guys,
>>> >>>>
>>> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>> This
>>> >>> bug
>>> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>>> and
>>> >>>> decided to revert this ticket. I will do it today (after TC bot
>>> visa)
>>> >> if
>>> >>>> there are no objections.
>>> >>>>
>>> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>> >>>>
>>> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>>> [hidden email]>:
>>> >>>>
>>> >>>>> Guys,
>>> >>>>>
>>> >>>>> During internal testing, we've found a critical bug with
>>> >>>>> discovery (cluster falls apart if two nodes segmented
>>> sequentially).
>>> >>> This
>>> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>> >>>>> before release. Under investigation now. I'll let you know when we
>>> get
>>> >>>>> something.
>>> >>>>>
>>> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>>> >>>>>
>>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>> of
>>> >>> the
>>> >>>>>> message factory in 2.9 and go for the runtime message generation
>>> in
>>> >>> later
>>> >>>>>> release, or keep the code clean and fix the regression in the next
>>> >>> releases?
>>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>> >>>>>> straightforward and does not change the semantics, just skips the
>>> >>> factory
>>> >>>>>> closures for certain messages.
>>> >>>>>>
>>> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>>> all
>>> >>>>>> workloads (I didn't get any significant drops during
>>> benchmarking).
>>> >> So
>>> >>>>>> I prefer the runtime generation in later releases.
>>> >>>>>>
>>> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>> >>>>>> <[hidden email]> wrote:
>>> >>>>>>>
>>> >>>>>>> Alexey, Andrey, Igniters,
>>> >>>>>>>
>>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>> of
>>> >>> the
>>> >>>>>> message factory in 2.9 and go for the runtime message generation
>>> in
>>> >>> later
>>> >>>>>> release, or keep the code clean and fix the regression in the next
>>> >>> releases?
>>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>> >>>>>> straightforward and does not change the semantics, just skips the
>>> >>> factory
>>> >>>>>> closures for certain messages.
>>> >>>>>>>
>>> >>>>>>> Personally, I would prefer fixing the regression given that we
>>> also
>>> >>>>>> introduced tracing in this release.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>> >> [hidden email]
>>> >>>> :
>>> >>>>>>>>
>>> >>>>>>>> Alexey,
>>> >>>>>>>>
>>> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>>> ed52559eb95
>>> >>> and
>>> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
>>> average on
>>> >>> our
>>> >>>>>> environment (about the same results we got benchmarking 65c30ec6
>>> vs
>>> >>>>>> 0606f03d). We've made 24 runs for each commit of
>>> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>>> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers,
>>> 5
>>> >>> clients
>>> >>>>>> 50 threads each. Yardstick results for this configuration:
>>> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>> >>>>>>>>
>>> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>> >>>>>> [hidden email]>:
>>> >>>>>>>>>
>>> >>>>>>>>> Hi Everyone,
>>> >>>>>>>>>
>>> >>>>>>>>> I posted an instruction on how to publish the docs on
>>> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
>>> can
>>> >>>>>> update the docs by following the instruction. Unfortunately, I
>>> won't
>>> >> be
>>> >>>>>> able to spend any time on this project any longer. You can send
>>> your
>>> >>> pull
>>> >>>>>> requests and questions about the documentation to Denis Magda.
>>> >>>>>>>>>
>>> >>>>>>>>> -Artem
>>> >>>>>>>>>
>>> >>>>>>>>> [1] :
>>> >>>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>> >>>>>> [hidden email]> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> Alexey,
>>> >>>>>>>>>>
>>> >>>>>>>>>> I've tried to play with message factories locally, but
>>> >>>>>> unfortunately, I
>>> >>>>>>>>>> cannot spot the difference between old and new implementation
>>> in
>>> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>> >>>>>> MessageFactoryImpl
>>> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>> >>> branch
>>> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>> >> compatible
>>> >>>>>> with the
>>> >>>>>>>>>> new metrics as we still use the register() mechanics).
>>> >>>>>>>>>>
>>> >>>>>>>>>> Can you check if this change makes any difference
>>> >> performance-wise
>>> >>>>>> in your
>>> >>>>>>>>>> environment? If yes, we can go with runtime code generation in
>>> >> the
>>> >>>>>> long
>>> >>>>>>>>>> term: register classes and generate a dynamic message factory
>>> >> with
>>> >>>>>> a switch
>>> >>>>>>>>>> statement once all messages are registered (not in 2.9 though,
>>> >>>>>> obviously).
>>> >>>>>>>>>>
>>> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>> >>> [hidden email]
>>> >>>>>>> :
>>> >>>>>>>>>>
>>> >>>>>>>>>>> Hello guys,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
>>> it
>>> >>>>>> reduced the
>>> >>>>>>>>>>> drop, but not completely removed it.
>>> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the patch?
>>> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>>> against
>>> >>>>>> 2.8.1
>>> >>>>>>>>>>> release on your environment?
>>> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>>> left,
>>> >>>>>> it's close
>>> >>>>>>>>>>> to measurement error and I think such a drop is not a
>>> >>>>>> showstopper. Guys,
>>> >>>>>>>>>>> WDYT?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>> >> driver
>>> >>>>>> between 2.8
>>> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>> >> should
>>> >>>>>> be
>>> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>> >>> message
>>> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it if
>>> >>>>>> there is no
>>> >>>>>>>>>>> objection.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>> >> [hidden email]
>>> >>>> :
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Alexey,
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>> >> this
>>> >>>>>> issue is
>>> >>>>>>>>>>>> related to the new master key change functionality which
>>> >>>>>> haven't been
>>> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>>> >> to
>>> >>>>>> the
>>> >>>>>>>>>>>> release branch.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>> >>>>>> [hidden email]>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Hello, Igniters.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>> >> [1]
>>> >>>>>> into the
>>> >>>>>>>>>>> 2.9
>>> >>>>>>>>>>>> release
>>> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>>> >>> in
>>> >>>>>> wrong
>>> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>> >>>>>>>>>>>>> [2]
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>
>>> >>>
>>> >>
>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>> >>>>>>>>>>> [hidden email]>
>>> >>>>>>>>>>>> написал(а):
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>> >>>>>> anything out of
>>> >>>>>>>>>>>> the
>>> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>> >>>>>> right now),
>>> >>>>>>>>>>> so
>>> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>> >>>>>> switch
>>> >>>>>>>>>>> approach
>>> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>> >>> the
>>> >>>>>> Ignite
>>> >>>>>>>>>>>> messages
>>> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>> >>> me
>>> >>>>>> explore
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>>>>> code a bit.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>> >> the
>>> >>>>>>>>>>> performance.
>>> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>> >>>>>> virtual call
>>> >>>>>>>>>>>>>> should not make that much of a difference given the
>>> >> amount
>>> >>>>>> of other
>>> >>>>>>>>>>>> work
>>> >>>>>>>>>>>>>> happening during the message processing.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>> >>>>>> [hidden email]
>>> >>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>> >>> results.
>>> >>>>>>>>>>> Actually,
>>> >>>>>>>>>>>> we
>>> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>> >>> between
>>> >>>>>> e6a7f93
>>> >>>>>>>>>>>> (first
>>> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>> >>>>>> 6592dfa5 (last
>>> >>>>>>>>>>>> commit in
>>> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>> >>>>>> commits.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>> >> a
>>> >>>>>> drop
>>> >>>>>>>>>>> between
>>> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>> >>>>>> reverted
>>> >>>>>>>>>>>> IGNITE-13060
>>> >>>>>>>>>>>>>>> now and performance looks the same)
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>> >>>>>> related to
>>> >>>>>>>>>>>> drop
>>> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>> >>>>>> problem, and
>>> >>>>>>>>>>> we
>>> >>>>>>>>>>>> can
>>> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>> >> leave
>>> >>>>>> it as is?
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>> >>>>>> refactoring)?
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>> >>>>>>>>>>>> [hidden email]
>>> >>>>>>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Alexey,
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>> >>>>>> incorrect fix
>>> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>> >>>>>> [1], so it
>>> >>>>>>>>>>>> cannot be
>>> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>> >> work
>>> >>>>>> with fix
>>> >>>>>>>>>>>> versions
>>> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>> >>>>>> versions.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> --AG
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> [1]
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>
>>> >>>
>>> >>
>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>> >>>>>>>>>>>> [hidden email]
>>> >>>>>>>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>> >>>>>>>>>>> [hidden email]
>>> >>>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Guys,
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>> >>>>>> IGNITE-12568
>>> >>>>>>>>>>>> (reverted
>>> >>>>>>>>>>>>>>>>>> it
>>> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>> >>>>>> paths, to
>>> >>>>>>>>>>> trace
>>> >>>>>>>>>>>>>>>>>> these
>>> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>> >>> here.
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>> >>> switch/case
>>> >>>>>> block was
>>> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>> >>>>>> message factory
>>> >>>>>>>>>>>> is on
>>> >>>>>>>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>> >> impact
>>> >>>>>> on total
>>> >>>>>>>>>>>>>>>>>> performance.
>>> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>> >>>>>> microbenchmarks,
>>> >>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>>>>> found
>>> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>> >>>>>> about 30-35%
>>> >>>>>>>>>>>> faster
>>> >>>>>>>>>>>>>>>>>> than
>>> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>> >>> can
>>> >>>>>>>>>>> effectively
>>> >>>>>>>>>>>>>>>>>> inline
>>> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>> >> suppliers
>>> >>>>>> relatively
>>> >>>>>>>>>>>> heavy
>>> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>> >>>>>> rewrite the
>>> >>>>>>>>>>> code
>>> >>>>>>>>>>>>>>>>>> using
>>> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>> >>> interface
>>> >>>>>> (to
>>> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>> >>>>>> but it gives
>>> >>>>>>>>>>>> back
>>> >>>>>>>>>>>>>>>>>> only
>>> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>> >> looks
>>> >>>>>> ugly
>>> >>>>>>>>>>>> (lambdas
>>> >>>>>>>>>>>>>>>>>> can't
>>> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>> >>>>>> optimize the
>>> >>>>>>>>>>>> current
>>> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>> >>>>>> Andrey Gura,
>>> >>>>>>>>>>> as
>>> >>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>> >>> about
>>> >>>>>>>>>>>> optimization?
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>> >>>>>> some metrics
>>> >>>>>>>>>>>>>>>>>> already
>>> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>> >>>>>> factory
>>> >>>>>>>>>>>>>>>>>> implementation
>>> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Alexey,
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>> >>>>>> already released
>>> >>>>>>>>>>>> in
>>> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>> >>>>>> only present
>>> >>>>>>>>>>>> in Ignite
>>> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>> >>>>>> metrics
>>> >>>>>>>>>>>> created for
>>> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>> >>>>>> the release
>>> >>>>>>>>>>>> and deal
>>> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >>
>>>
>>>

[dmagda]

@Alex Plehanov <[hidden email]>,

If it still makes sense and not too late, you can cherry-pick the commit
with the new docs into the 2.9 branch:
https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191

Anyway, it's not crucial as long as we agreed to make an exception for this
release. The docs were already published to the website and assembled from
the master. Once the vote passes, I'll make them visible and traceable from
the website menus.

-
Denis


On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <[hidden email]>
wrote:

> Saikat,
>
> The plan sounds good to me! Do you have an idea for the timeline of the
> module releases? Let me know if I can help in any way.
>
> Also, I was looking into the modularization IEP and noticed that OSGI
> module is discussed to be moved to the extensions, but there is no
> corresponding ticket. Should I create one? I will be happy to help with
> moving this module to extensions.
>
> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]>:
>
> > Hi Alexey,
> >
> > All the modules as planned in IEP-36
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> > are migrated to ignite-extensions repository.
> >
> > We can definitely plan to release ignite-extensions modules.
> >
> > I have a pending PR related to the kafka module and the PR is in review
> > https://github.com/apache/ignite/pull/8222 but its corresponding PR has
> > been merged in ignite-extensions repository.
> >
> > My thoughts / action items for the migration efforts and releases were
> > as follows:
> >
> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> > 2. Fix the upload nightly packages task for ignite-core to help fix the
> > travis build failure in ignite-extensions repository.
> > 3. Review and update the README.md files for the ignite-extensions
> modules
> > as required.
> > 4. Update the docs for ignite-extensions modules in ignite-website.
> > 5. Release each module separately and share updates.
> >
> > Please let me know if you have feedback.
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]> wrote:
> >
> >> It seems that gitbox.apache.org is not available on CI/CD.
> >>
> >> I will try to update VCS to GitHub.
> >>
> >>
> >>
> >>
> >>
> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
> >> wrote:
> >> >
> >> > Guys,
> >> >
> >> > I've tried to build release candidate using TC [1]. But:
> >> > 1. I can't choose a branch in this TC task (there is no such field).
> >> > 2. On the "default" branch I got an error: Failed to collect changes,
> >> > error: List remote refs failed:
> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> connect
> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >> >
> >> > Is there some problem with this TC task? What am I doing wrong?
> >> >
> >> > [1] :
> >> >
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >> >
> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >> [hidden email]>:
> >> >
> >> >> Saikat, Nikolay,
> >> >>
> >> >> We have migrated a bunch of modules to ignite-extensions recently.
> >> Given
> >> >> that these modules will not be available in Ignite 2.9 anymore (will
> >> >> they?), should we also plan to release the extensions?
> >> >>
> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]
> >:
> >> >>
> >> >>> Igniters,
> >> >>>
> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
> review
> >> >> it,
> >> >>> please?
> >> >>>
> >> >>> [1]: https://github.com/apache/ignite/pull/8273
> >> >>>
> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> [hidden email]
> >> >:
> >> >>>
> >> >>>> Guys,
> >> >>>>
> >> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
> >> This
> >> >>> bug
> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
> >> and
> >> >>>> decided to revert this ticket. I will do it today (after TC bot
> visa)
> >> >> if
> >> >>>> there are no objections.
> >> >>>>
> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >> >>>>
> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> [hidden email]
> >> >:
> >> >>>>
> >> >>>>> Guys,
> >> >>>>>
> >> >>>>> During internal testing, we've found a critical bug with
> >> >>>>> discovery (cluster falls apart if two nodes segmented
> sequentially).
> >> >>> This
> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >> >>>>> before release. Under investigation now. I'll let you know when we
> >> get
> >> >>>>> something.
> >> >>>>>
> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
> >> >>>>>
> >> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
> of
> >> >>> the
> >> >>>>>> message factory in 2.9 and go for the runtime message generation
> in
> >> >>> later
> >> >>>>>> release, or keep the code clean and fix the regression in the
> next
> >> >>> releases?
> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >> >>>>>> straightforward and does not change the semantics, just skips the
> >> >>> factory
> >> >>>>>> closures for certain messages.
> >> >>>>>>
> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
> all
> >> >>>>>> workloads (I didn't get any significant drops during
> benchmarking).
> >> >> So
> >> >>>>>> I prefer the runtime generation in later releases.
> >> >>>>>>
> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >> >>>>>> <[hidden email]> wrote:
> >> >>>>>>>
> >> >>>>>>> Alexey, Andrey, Igniters,
> >> >>>>>>>
> >> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
> of
> >> >>> the
> >> >>>>>> message factory in 2.9 and go for the runtime message generation
> in
> >> >>> later
> >> >>>>>> release, or keep the code clean and fix the regression in the
> next
> >> >>> releases?
> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >> >>>>>> straightforward and does not change the semantics, just skips the
> >> >>> factory
> >> >>>>>> closures for certain messages.
> >> >>>>>>>
> >> >>>>>>> Personally, I would prefer fixing the regression given that we
> >> also
> >> >>>>>> introduced tracing in this release.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >> >> [hidden email]
> >> >>>> :
> >> >>>>>>>>
> >> >>>>>>>> Alexey,
> >> >>>>>>>>
> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> ed52559eb95
> >> >>> and
> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
> average
> >> on
> >> >>> our
> >> >>>>>> environment (about the same results we got benchmarking 65c30ec6
> vs
> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> servers, 5
> >> >>> clients
> >> >>>>>> 50 threads each. Yardstick results for this configuration:
> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >> >>>>>>>>
> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >> >>>>>> [hidden email]>:
> >> >>>>>>>>>
> >> >>>>>>>>> Hi Everyone,
> >> >>>>>>>>>
> >> >>>>>>>>> I posted an instruction on how to publish the docs on
> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
> >> can
> >> >>>>>> update the docs by following the instruction. Unfortunately, I
> >> won't
> >> >> be
> >> >>>>>> able to spend any time on this project any longer. You can send
> >> your
> >> >>> pull
> >> >>>>>> requests and questions about the documentation to Denis Magda.
> >> >>>>>>>>>
> >> >>>>>>>>> -Artem
> >> >>>>>>>>>
> >> >>>>>>>>> [1] :
> >> >>>>>>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >> >>>>>>>>>
> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >> >>>>>> [hidden email]> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> Alexey,
> >> >>>>>>>>>>
> >> >>>>>>>>>> I've tried to play with message factories locally, but
> >> >>>>>> unfortunately, I
> >> >>>>>>>>>> cannot spot the difference between old and new implementation
> >> in
> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >> >>>>>> MessageFactoryImpl
> >> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
> >> >>> branch
> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >> >> compatible
> >> >>>>>> with the
> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
> >> >>>>>>>>>>
> >> >>>>>>>>>> Can you check if this change makes any difference
> >> >> performance-wise
> >> >>>>>> in your
> >> >>>>>>>>>> environment? If yes, we can go with runtime code generation
> in
> >> >> the
> >> >>>>>> long
> >> >>>>>>>>>> term: register classes and generate a dynamic message factory
> >> >> with
> >> >>>>>> a switch
> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
> though,
> >> >>>>>> obviously).
> >> >>>>>>>>>>
> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >> >>> [hidden email]
> >> >>>>>>> :
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Hello guys,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
> it
> >> >>>>>> reduced the
> >> >>>>>>>>>>> drop, but not completely removed it.
> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> patch?
> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> against
> >> >>>>>> 2.8.1
> >> >>>>>>>>>>> release on your environment?
> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> left,
> >> >>>>>> it's close
> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
> >> >>>>>> showstopper. Guys,
> >> >>>>>>>>>>> WDYT?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >> >> driver
> >> >>>>>> between 2.8
> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >> >> should
> >> >>>>>> be
> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
> >> >>> message
> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
> if
> >> >>>>>> there is no
> >> >>>>>>>>>>> objection.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >> >> [hidden email]
> >> >>>> :
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >> >> this
> >> >>>>>> issue is
> >> >>>>>>>>>>>> related to the new master key change functionality which
> >> >>>>>> haven't been
> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
> >> >> to
> >> >>>>>> the
> >> >>>>>>>>>>>> release branch.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >> >>>>>> [hidden email]>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Hello, Igniters.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >> >> [1]
> >> >>>>>> into the
> >> >>>>>>>>>>> 2.9
> >> >>>>>>>>>>>> release
> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
> >> >>> in
> >> >>>>>> wrong
> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >> >>>>>>>>>>>>> [2]
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >> >>>>>>>>>>> [hidden email]>
> >> >>>>>>>>>>>> написал(а):
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >> >>>>>> anything out of
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
> >> >>>>>> right now),
> >> >>>>>>>>>>> so
> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
> >> >>>>>> switch
> >> >>>>>>>>>>> approach
> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
> >> >>> the
> >> >>>>>> Ignite
> >> >>>>>>>>>>>> messages
> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
> >> >>> me
> >> >>>>>> explore
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>>>>> code a bit.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >> >> the
> >> >>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
> >> >>>>>> virtual call
> >> >>>>>>>>>>>>>> should not make that much of a difference given the
> >> >> amount
> >> >>>>>> of other
> >> >>>>>>>>>>>> work
> >> >>>>>>>>>>>>>> happening during the message processing.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >> >>>>>> [hidden email]
> >> >>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >> >>> results.
> >> >>>>>>>>>>> Actually,
> >> >>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >> >>> between
> >> >>>>>> e6a7f93
> >> >>>>>>>>>>>> (first
> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >> >>>>>> 6592dfa5 (last
> >> >>>>>>>>>>>> commit in
> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >> >>>>>> commits.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
> >> >> a
> >> >>>>>> drop
> >> >>>>>>>>>>> between
> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >> >>>>>> reverted
> >> >>>>>>>>>>>> IGNITE-13060
> >> >>>>>>>>>>>>>>> now and performance looks the same)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >> >>>>>> related to
> >> >>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >> >>>>>> problem, and
> >> >>>>>>>>>>> we
> >> >>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >> >> leave
> >> >>>>>> it as is?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >> >>>>>> refactoring)?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >> >>>>>>>>>>>> [hidden email]
> >> >>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
> >> >>>>>> incorrect fix
> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
> >> >>>>>> [1], so it
> >> >>>>>>>>>>>> cannot be
> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >> >> work
> >> >>>>>> with fix
> >> >>>>>>>>>>>> versions
> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >> >>>>>> versions.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> --AG
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> [1]
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >> >>>>>>>>>>>> [hidden email]
> >> >>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >> >>>>>>>>>>> [hidden email]
> >> >>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Guys,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >> >>>>>> IGNITE-12568
> >> >>>>>>>>>>>> (reverted
> >> >>>>>>>>>>>>>>>>>> it
> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >> >>>>>> paths, to
> >> >>>>>>>>>>> trace
> >> >>>>>>>>>>>>>>>>>> these
> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >> >>> here.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >> >>> switch/case
> >> >>>>>> block was
> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >> >>>>>> message factory
> >> >>>>>>>>>>>> is on
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >> >> impact
> >> >>>>>> on total
> >> >>>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >> >>>>>> microbenchmarks,
> >> >>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>> found
> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >> >>>>>> about 30-35%
> >> >>>>>>>>>>>> faster
> >> >>>>>>>>>>>>>>>>>> than
> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
> >> >>> can
> >> >>>>>>>>>>> effectively
> >> >>>>>>>>>>>>>>>>>> inline
> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >> >> suppliers
> >> >>>>>> relatively
> >> >>>>>>>>>>>> heavy
> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >> >>>>>> rewrite the
> >> >>>>>>>>>>> code
> >> >>>>>>>>>>>>>>>>>> using
> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >> >>> interface
> >> >>>>>> (to
> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
> >> >>>>>> but it gives
> >> >>>>>>>>>>>> back
> >> >>>>>>>>>>>>>>>>>> only
> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >> >> looks
> >> >>>>>> ugly
> >> >>>>>>>>>>>> (lambdas
> >> >>>>>>>>>>>>>>>>>> can't
> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >> >>>>>> optimize the
> >> >>>>>>>>>>>> current
> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >> >>>>>> Andrey Gura,
> >> >>>>>>>>>>> as
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >> >>> about
> >> >>>>>>>>>>>> optimization?
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
> >> >>>>>> some metrics
> >> >>>>>>>>>>>>>>>>>> already
> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
> >> >>>>>> factory
> >> >>>>>>>>>>>>>>>>>> implementation
> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >> >>>>>> already released
> >> >>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >> >>>>>> only present
> >> >>>>>>>>>>>> in Ignite
> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >> >>>>>> metrics
> >> >>>>>>>>>>>> created for
> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
> >> >>>>>> the release
> >> >>>>>>>>>>>> and deal
> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> >>
> >>
> >>
>

[Nikolay Izhikov-2]

Hello, Igniters.

I prepared a patch [1] for blocker ticket [2] - «Server node fail and stops in case wrong datatype put in indexed field»
Can someone, please, help me with the review?

[1] https://github.com/apache/ignite/pull/8330
[2] https://issues.apache.org/jira/browse/IGNITE-13553

==================

I propose to include several minor tickets to the scope of the 2.9 that increase Ignite User Experience

CMD tools improvements:

IGNITE-13488 - Command to print metric value
IGNITE-13426 - Command to print system view content
IGNITE-13422 - Parameter to explicitly enable experimental commands

IGNITE-13380 - Output IgniteSystemProperties via ignite.sh

New system views:

IGNITE-13409 Metastorage and DistributedMetastorage viewы.
IGNITE-13408 BinaryMetadata view.

> 9 окт. 2020 г., в 04:04, Denis Magda <[hidden email]> написал(а):
>
> @Alex Plehanov <[hidden email]>,
>
> If it still makes sense and not too late, you can cherry-pick the commit
> with the new docs into the 2.9 branch:
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
>
> Anyway, it's not crucial as long as we agreed to make an exception for this
> release. The docs were already published to the website and assembled from
> the master. Once the vote passes, I'll make them visible and traceable from
> the website menus.
>
> -
> Denis
>
>
> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <[hidden email]>
> wrote:
>
>> Saikat,
>>
>> The plan sounds good to me! Do you have an idea for the timeline of the
>> module releases? Let me know if I can help in any way.
>>
>> Also, I was looking into the modularization IEP and noticed that OSGI
>> module is discussed to be moved to the extensions, but there is no
>> corresponding ticket. Should I create one? I will be happy to help with
>> moving this module to extensions.
>>
>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]>:
>>
>>> Hi Alexey,
>>>
>>> All the modules as planned in IEP-36
>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>>> are migrated to ignite-extensions repository.
>>>
>>> We can definitely plan to release ignite-extensions modules.
>>>
>>> I have a pending PR related to the kafka module and the PR is in review
>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>>> been merged in ignite-extensions repository.
>>>
>>> My thoughts / action items for the migration efforts and releases were
>>> as follows:
>>>
>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>>> 2. Fix the upload nightly packages task for ignite-core to help fix the
>>> travis build failure in ignite-extensions repository.
>>> 3. Review and update the README.md files for the ignite-extensions
>> modules
>>> as required.
>>> 4. Update the docs for ignite-extensions modules in ignite-website.
>>> 5. Release each module separately and share updates.
>>>
>>> Please let me know if you have feedback.
>>>
>>> Regards,
>>> Saikat
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]> wrote:
>>>
>>>> It seems that gitbox.apache.org is not available on CI/CD.
>>>>
>>>> I will try to update VCS to GitHub.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
>>>> wrote:
>>>>>
>>>>> Guys,
>>>>>
>>>>> I've tried to build release candidate using TC [1]. But:
>>>>> 1. I can't choose a branch in this TC task (there is no such field).
>>>>> 2. On the "default" branch I got an error: Failed to collect changes,
>>>>> error: List remote refs failed:
>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> connect
>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>>>>
>>>>> Is there some problem with this TC task? What am I doing wrong?
>>>>>
>>>>> [1] :
>>>>>
>>>>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>>>>
>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>>> [hidden email]>:
>>>>>
>>>>>> Saikat, Nikolay,
>>>>>>
>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
>>>> Given
>>>>>> that these modules will not be available in Ignite 2.9 anymore (will
>>>>>> they?), should we also plan to release the extensions?
>>>>>>
>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <[hidden email]
>>> :
>>>>>>
>>>>>>> Igniters,
>>>>>>>
>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
>> review
>>>>>> it,
>>>>>>> please?
>>>>>>>
>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
>>>>>>>
>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>> [hidden email]
>>>>> :
>>>>>>>
>>>>>>>> Guys,
>>>>>>>>
>>>>>>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>>> This
>>>>>>> bug
>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>>>> and
>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
>> visa)
>>>>>> if
>>>>>>>> there are no objections.
>>>>>>>>
>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>>>>>>>
>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>> [hidden email]
>>>>> :
>>>>>>>>
>>>>>>>>> Guys,
>>>>>>>>>
>>>>>>>>> During internal testing, we've found a critical bug with
>>>>>>>>> discovery (cluster falls apart if two nodes segmented
>> sequentially).
>>>>>>> This
>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>>>>>>>> before release. Under investigation now. I'll let you know when we
>>>> get
>>>>>>>>> something.
>>>>>>>>>
>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>>>>>>>>>
>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>> of
>>>>>>> the
>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>> in
>>>>>>> later
>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> next
>>>>>>> releases?
>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>> straightforward and does not change the semantics, just skips the
>>>>>>> factory
>>>>>>>>>> closures for certain messages.
>>>>>>>>>>
>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>> all
>>>>>>>>>> workloads (I didn't get any significant drops during
>> benchmarking).
>>>>>> So
>>>>>>>>>> I prefer the runtime generation in later releases.
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>>>>>>>>> <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Alexey, Andrey, Igniters,
>>>>>>>>>>>
>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>> of
>>>>>>> the
>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>> in
>>>>>>> later
>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> next
>>>>>>> releases?
>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>> straightforward and does not change the semantics, just skips the
>>>>>>> factory
>>>>>>>>>> closures for certain messages.
>>>>>>>>>>>
>>>>>>>>>>> Personally, I would prefer fixing the regression given that we
>>>> also
>>>>>>>>>> introduced tracing in this release.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>>>>> [hidden email]
>>>>>>>> :
>>>>>>>>>>>>
>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>
>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>> ed52559eb95
>>>>>>> and
>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
>> average
>>>> on
>>>>>>> our
>>>>>>>>>> environment (about the same results we got benchmarking 65c30ec6
>> vs
>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>> servers, 5
>>>>>>> clients
>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>>>>>>>>>>>
>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>>>>>>>>> [hidden email]>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
>>>> can
>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
>>>> won't
>>>>>> be
>>>>>>>>>> able to spend any time on this project any longer. You can send
>>>> your
>>>>>>> pull
>>>>>>>>>> requests and questions about the documentation to Denis Magda.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Artem
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] :
>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>>>>>>>>> [hidden email]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've tried to play with message factories locally, but
>>>>>>>>>> unfortunately, I
>>>>>>>>>>>>>> cannot spot the difference between old and new implementation
>>>> in
>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>>>>>>>>> MessageFactoryImpl
>>>>>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>>>>>> branch
>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>>>>> compatible
>>>>>>>>>> with the
>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you check if this change makes any difference
>>>>>> performance-wise
>>>>>>>>>> in your
>>>>>>>>>>>>>> environment? If yes, we can go with runtime code generation
>> in
>>>>>> the
>>>>>>>>>> long
>>>>>>>>>>>>>> term: register classes and generate a dynamic message factory
>>>>>> with
>>>>>>>>>> a switch
>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
>> though,
>>>>>>>>>> obviously).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>>>>>> [hidden email]
>>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
>> it
>>>>>>>>>> reduced the
>>>>>>>>>>>>>>> drop, but not completely removed it.
>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>> patch?
>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>> against
>>>>>>>>>> 2.8.1
>>>>>>>>>>>>>>> release on your environment?
>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>> left,
>>>>>>>>>> it's close
>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
>>>>>>>>>> showstopper. Guys,
>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>>>>> driver
>>>>>>>>>> between 2.8
>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>>>>> should
>>>>>>>>>> be
>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>>>>>> message
>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
>> if
>>>>>>>>>> there is no
>>>>>>>>>>>>>>> objection.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>>>>> [hidden email]
>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>>>>> this
>>>>>>>>>> issue is
>>>>>>>>>>>>>>>> related to the new master key change functionality which
>>>>>>>>>> haven't been
>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>>>>> release branch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello, Igniters.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>>>>> [1]
>>>>>>>>>> into the
>>>>>>>>>>>>>>> 2.9
>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>>>>>>> in
>>>>>>>>>> wrong
>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>>>>>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>>>>>>>>> anything out of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>>>>>>>>> right now),
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>>>>>>>>> switch
>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>>>>>> the
>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>>>>>> me
>>>>>>>>>> explore
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> code a bit.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>>>>> the
>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>>>>>>>>> virtual call
>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
>>>>>> amount
>>>>>>>>>> of other
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> happening during the message processing.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>>>>>> results.
>>>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>>>>>> between
>>>>>>>>>> e6a7f93
>>>>>>>>>>>>>>>> (first
>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>>>>>>>>> 6592dfa5 (last
>>>>>>>>>>>>>>>> commit in
>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>>>>>>>>> commits.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>>>>> a
>>>>>>>>>> drop
>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>>>>>>>>> reverted
>>>>>>>>>>>>>>>> IGNITE-13060
>>>>>>>>>>>>>>>>>>> now and performance looks the same)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>>>>>>>>> related to
>>>>>>>>>>>>>>>> drop
>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>>>>>>>>> problem, and
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>>>>> leave
>>>>>>>>>> it as is?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>>>>>>>>> refactoring)?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>>>>>>>>> incorrect fix
>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>>>>>>>>> [1], so it
>>>>>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>>>>> work
>>>>>>>>>> with fix
>>>>>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>>>>>>>>> versions.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --AG
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>>>>>>>> IGNITE-12568
>>>>>>>>>>>>>>>> (reverted
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>>>>>>>>> paths, to
>>>>>>>>>>>>>>> trace
>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>>>>>> switch/case
>>>>>>>>>> block was
>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>>>>>>>>> message factory
>>>>>>>>>>>>>>>> is on
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>>>>> impact
>>>>>>>>>> on total
>>>>>>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>>>>>>>>> microbenchmarks,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>>>>>>>>> about 30-35%
>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>>>>>> can
>>>>>>>>>>>>>>> effectively
>>>>>>>>>>>>>>>>>>>>>> inline
>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>>>>> suppliers
>>>>>>>>>> relatively
>>>>>>>>>>>>>>>> heavy
>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>>>>>>>>> rewrite the
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>>>>>> interface
>>>>>>>>>> (to
>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>>>>>>>>> but it gives
>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>>>>> looks
>>>>>>>>>> ugly
>>>>>>>>>>>>>>>> (lambdas
>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>>>>>>>>> optimize the
>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>>>>>>>>> Andrey Gura,
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>>>>>> about
>>>>>>>>>>>>>>>> optimization?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>>>>>>>>> some metrics
>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>>>>>>>>> factory
>>>>>>>>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>>>>>>>>> already released
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>>>>>>>>> only present
>>>>>>>>>>>>>>>> in Ignite
>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>>>>>>>>> metrics
>>>>>>>>>>>>>>>> created for
>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>>>>>>>>> the release
>>>>>>>>>>>>>>>> and deal
>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>

[dmagda]

Nikolay,

re: the minor improvements. I'm not against of including those if we can
prepare the docs before the vote starts. Presently, the docs are "frozen"
for the 2.9 release, but I can scratch some time and take part in the docs
review the next week.

-
Denis


On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <[hidden email]> wrote:

> Hello, Igniters.
>
> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> stops in case wrong datatype put in indexed field»
> Can someone, please, help me with the review?
>
> [1] https://github.com/apache/ignite/pull/8330
> [2] https://issues.apache.org/jira/browse/IGNITE-13553
>
> ==================
>
> I propose to include several minor tickets to the scope of the 2.9 that
> increase Ignite User Experience
>
> CMD tools improvements:
>
> IGNITE-13488 - Command to print metric value
> IGNITE-13426 - Command to print system view content
> IGNITE-13422 - Parameter to explicitly enable experimental commands
>
> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>
> New system views:
>
> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> IGNITE-13408 BinaryMetadata view.
>
> > 9 окт. 2020 г., в 04:04, Denis Magda <[hidden email]> написал(а):
> >
> > @Alex Plehanov <[hidden email]>,
> >
> > If it still makes sense and not too late, you can cherry-pick the commit
> > with the new docs into the 2.9 branch:
> >
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> >
> > Anyway, it's not crucial as long as we agreed to make an exception for
> this
> > release. The docs were already published to the website and assembled
> from
> > the master. Once the vote passes, I'll make them visible and traceable
> from
> > the website menus.
> >
> > -
> > Denis
> >
> >
> > On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> [hidden email]>
> > wrote:
> >
> >> Saikat,
> >>
> >> The plan sounds good to me! Do you have an idea for the timeline of the
> >> module releases? Let me know if I can help in any way.
> >>
> >> Also, I was looking into the modularization IEP and noticed that OSGI
> >> module is discussed to be moved to the extensions, but there is no
> >> corresponding ticket. Should I create one? I will be happy to help with
> >> moving this module to extensions.
> >>
> >> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]>:
> >>
> >>> Hi Alexey,
> >>>
> >>> All the modules as planned in IEP-36
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> >>> are migrated to ignite-extensions repository.
> >>>
> >>> We can definitely plan to release ignite-extensions modules.
> >>>
> >>> I have a pending PR related to the kafka module and the PR is in review
> >>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
> has
> >>> been merged in ignite-extensions repository.
> >>>
> >>> My thoughts / action items for the migration efforts and releases were
> >>> as follows:
> >>>
> >>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> >>> 2. Fix the upload nightly packages task for ignite-core to help fix the
> >>> travis build failure in ignite-extensions repository.
> >>> 3. Review and update the README.md files for the ignite-extensions
> >> modules
> >>> as required.
> >>> 4. Update the docs for ignite-extensions modules in ignite-website.
> >>> 5. Release each module separately and share updates.
> >>>
> >>> Please let me know if you have feedback.
> >>>
> >>> Regards,
> >>> Saikat
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]>
> wrote:
> >>>
> >>>> It seems that gitbox.apache.org is not available on CI/CD.
> >>>>
> >>>> I will try to update VCS to GitHub.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
> >>>> wrote:
> >>>>>
> >>>>> Guys,
> >>>>>
> >>>>> I've tried to build release candidate using TC [1]. But:
> >>>>> 1. I can't choose a branch in this TC task (there is no such field).
> >>>>> 2. On the "default" branch I got an error: Failed to collect changes,
> >>>>> error: List remote refs failed:
> >>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> >>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> >> connect
> >>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> >>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >>>>>
> >>>>> Is there some problem with this TC task? What am I doing wrong?
> >>>>>
> >>>>> [1] :
> >>>>>
> >>>>
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >>>>>
> >>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >>>> [hidden email]>:
> >>>>>
> >>>>>> Saikat, Nikolay,
> >>>>>>
> >>>>>> We have migrated a bunch of modules to ignite-extensions recently.
> >>>> Given
> >>>>>> that these modules will not be available in Ignite 2.9 anymore (will
> >>>>>> they?), should we also plan to release the extensions?
> >>>>>>
> >>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> [hidden email]
> >>> :
> >>>>>>
> >>>>>>> Igniters,
> >>>>>>>
> >>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
> >> review
> >>>>>> it,
> >>>>>>> please?
> >>>>>>>
> >>>>>>> [1]: https://github.com/apache/ignite/pull/8273
> >>>>>>>
> >>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> >> [hidden email]
> >>>>> :
> >>>>>>>
> >>>>>>>> Guys,
> >>>>>>>>
> >>>>>>>> I've filled the ticket with reproducer [1] for the discovery bug.
> >>>> This
> >>>>>>> bug
> >>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
> >>>> and
> >>>>>>>> decided to revert this ticket. I will do it today (after TC bot
> >> visa)
> >>>>>> if
> >>>>>>>> there are no objections.
> >>>>>>>>
> >>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >>>>>>>>
> >>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> >> [hidden email]
> >>>>> :
> >>>>>>>>
> >>>>>>>>> Guys,
> >>>>>>>>>
> >>>>>>>>> During internal testing, we've found a critical bug with
> >>>>>>>>> discovery (cluster falls apart if two nodes segmented
> >> sequentially).
> >>>>>>> This
> >>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >>>>>>>>> before release. Under investigation now. I'll let you know when
> we
> >>>> get
> >>>>>>>>> something.
> >>>>>>>>>
> >>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
> >>>>>>>>>
> >>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
> >> of
> >>>>>>> the
> >>>>>>>>>> message factory in 2.9 and go for the runtime message generation
> >> in
> >>>>>>> later
> >>>>>>>>>> release, or keep the code clean and fix the regression in the
> >> next
> >>>>>>> releases?
> >>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >>>>>>>>>> straightforward and does not change the semantics, just skips
> the
> >>>>>>> factory
> >>>>>>>>>> closures for certain messages.
> >>>>>>>>>>
> >>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
> >> all
> >>>>>>>>>> workloads (I didn't get any significant drops during
> >> benchmarking).
> >>>>>> So
> >>>>>>>>>> I prefer the runtime generation in later releases.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >>>>>>>>>> <[hidden email]> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Alexey, Andrey, Igniters,
> >>>>>>>>>>>
> >>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
> >> of
> >>>>>>> the
> >>>>>>>>>> message factory in 2.9 and go for the runtime message generation
> >> in
> >>>>>>> later
> >>>>>>>>>> release, or keep the code clean and fix the regression in the
> >> next
> >>>>>>> releases?
> >>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >>>>>>>>>> straightforward and does not change the semantics, just skips
> the
> >>>>>>> factory
> >>>>>>>>>> closures for certain messages.
> >>>>>>>>>>>
> >>>>>>>>>>> Personally, I would prefer fixing the regression given that we
> >>>> also
> >>>>>>>>>> introduced tracing in this release.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >>>>>> [hidden email]
> >>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> >> ed52559eb95
> >>>>>>> and
> >>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> >> average
> >>>> on
> >>>>>>> our
> >>>>>>>>>> environment (about the same results we got benchmarking 65c30ec6
> >> vs
> >>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> >>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
> this
> >>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> >> servers, 5
> >>>>>>> clients
> >>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> >>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>>>>>>>>>>>
> >>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >>>>>>>>>> [hidden email]>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> >>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
> you
> >>>> can
> >>>>>>>>>> update the docs by following the instruction. Unfortunately, I
> >>>> won't
> >>>>>> be
> >>>>>>>>>> able to spend any time on this project any longer. You can send
> >>>> your
> >>>>>>> pull
> >>>>>>>>>> requests and questions about the documentation to Denis Magda.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Artem
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] :
> >>>>>>>>>>
> >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>>>>>>>>> [hidden email]> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've tried to play with message factories locally, but
> >>>>>>>>>> unfortunately, I
> >>>>>>>>>>>>>> cannot spot the difference between old and new
> implementation
> >>>> in
> >>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >>>>>>>>>> MessageFactoryImpl
> >>>>>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
> >>>>>>> branch
> >>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >>>>>> compatible
> >>>>>>>>>> with the
> >>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Can you check if this change makes any difference
> >>>>>> performance-wise
> >>>>>>>>>> in your
> >>>>>>>>>>>>>> environment? If yes, we can go with runtime code generation
> >> in
> >>>>>> the
> >>>>>>>>>> long
> >>>>>>>>>>>>>> term: register classes and generate a dynamic message
> factory
> >>>>>> with
> >>>>>>>>>> a switch
> >>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> >> though,
> >>>>>>>>>> obviously).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >>>>>>> [hidden email]
> >>>>>>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello guys,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
> >> it
> >>>>>>>>>> reduced the
> >>>>>>>>>>>>>>> drop, but not completely removed it.
> >>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> >> patch?
> >>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> >> against
> >>>>>>>>>> 2.8.1
> >>>>>>>>>>>>>>> release on your environment?
> >>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> >> left,
> >>>>>>>>>> it's close
> >>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> >>>>>>>>>> showstopper. Guys,
> >>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >>>>>> driver
> >>>>>>>>>> between 2.8
> >>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >>>>>> should
> >>>>>>>>>> be
> >>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
> >>>>>>> message
> >>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
> >> if
> >>>>>>>>>> there is no
> >>>>>>>>>>>>>>> objection.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >>>>>> [hidden email]
> >>>>>>>> :
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >>>>>> this
> >>>>>>>>>> issue is
> >>>>>>>>>>>>>>>> related to the new master key change functionality which
> >>>>>>>>>> haven't been
> >>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
> >>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> release branch.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >>>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello, Igniters.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >>>>>> [1]
> >>>>>>>>>> into the
> >>>>>>>>>>>>>>> 2.9
> >>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> fields
> >>>>>>> in
> >>>>>>>>>> wrong
> >>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>>>>>>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >>>>>>>>>> anything out of
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
> >>>>>>>>>> right now),
> >>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to the
> >>>>>>>>>> switch
> >>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
> >>>>>>> the
> >>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>> messages
> >>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
> >>>>>>> me
> >>>>>>>>>> explore
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> code a bit.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >>>>>> the
> >>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
> >>>>>>>>>> virtual call
> >>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> >>>>>> amount
> >>>>>>>>>> of other
> >>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>> happening during the message processing.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >>>>>>> results.
> >>>>>>>>>>>>>>> Actually,
> >>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >>>>>>> between
> >>>>>>>>>> e6a7f93
> >>>>>>>>>>>>>>>> (first
> >>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >>>>>>>>>> 6592dfa5 (last
> >>>>>>>>>>>>>>>> commit in
> >>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >>>>>>>>>> commits.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
> >>>>>> a
> >>>>>>>>>> drop
> >>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >>>>>>>>>> reverted
> >>>>>>>>>>>>>>>> IGNITE-13060
> >>>>>>>>>>>>>>>>>>> now and performance looks the same)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >>>>>>>>>> related to
> >>>>>>>>>>>>>>>> drop
> >>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >>>>>>>>>> problem, and
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >>>>>> leave
> >>>>>>>>>> it as is?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >>>>>>>>>> refactoring)?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>>>>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
> >>>>>>>>>> incorrect fix
> >>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
> >>>>>>>>>> [1], so it
> >>>>>>>>>>>>>>>> cannot be
> >>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >>>>>> work
> >>>>>>>>>> with fix
> >>>>>>>>>>>>>>>> versions
> >>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >>>>>>>>>> versions.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --AG
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>>>>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>>>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Guys,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>>>>>>>>> IGNITE-12568
> >>>>>>>>>>>>>>>> (reverted
> >>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >>>>>>>>>> paths, to
> >>>>>>>>>>>>>>> trace
> >>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >>>>>>> switch/case
> >>>>>>>>>> block was
> >>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >>>>>>>>>> message factory
> >>>>>>>>>>>>>>>> is on
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >>>>>> impact
> >>>>>>>>>> on total
> >>>>>>>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >>>>>>>>>> microbenchmarks,
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >>>>>>>>>> about 30-35%
> >>>>>>>>>>>>>>>> faster
> >>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
> >>>>>>> can
> >>>>>>>>>>>>>>> effectively
> >>>>>>>>>>>>>>>>>>>>>> inline
> >>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >>>>>> suppliers
> >>>>>>>>>> relatively
> >>>>>>>>>>>>>>>> heavy
> >>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >>>>>>>>>> rewrite the
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >>>>>>> interface
> >>>>>>>>>> (to
> >>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
> >>>>>>>>>> but it gives
> >>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >>>>>> looks
> >>>>>>>>>> ugly
> >>>>>>>>>>>>>>>> (lambdas
> >>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >>>>>>>>>> optimize the
> >>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >>>>>>>>>> Andrey Gura,
> >>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >>>>>>> about
> >>>>>>>>>>>>>>>> optimization?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
> >>>>>>>>>> some metrics
> >>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
> >>>>>>>>>> factory
> >>>>>>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >>>>>>>>>> already released
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >>>>>>>>>> only present
> >>>>>>>>>>>>>>>> in Ignite
> >>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >>>>>>>>>> metrics
> >>>>>>>>>>>>>>>> created for
> >>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
> >>>>>>>>>> the release
> >>>>>>>>>>>>>>>> and deal
> >>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
>
>

[Nikolay Izhikov-2]

Igniters,

IGNITE-13553 fixed and cherry-picked to 2.9.
We can continue with the release.

> 10 окт. 2020 г., в 00:00, Denis Magda <[hidden email]> написал(а):
>
> Nikolay,
>
> re: the minor improvements. I'm not against of including those if we can
> prepare the docs before the vote starts. Presently, the docs are "frozen"
> for the 2.9 release, but I can scratch some time and take part in the docs
> review the next week.
>
> -
> Denis
>
>
> On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <[hidden email]> wrote:
>
>> Hello, Igniters.
>>
>> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
>> stops in case wrong datatype put in indexed field»
>> Can someone, please, help me with the review?
>>
>> [1] https://github.com/apache/ignite/pull/8330
>> [2] https://issues.apache.org/jira/browse/IGNITE-13553
>>
>> ==================
>>
>> I propose to include several minor tickets to the scope of the 2.9 that
>> increase Ignite User Experience
>>
>> CMD tools improvements:
>>
>> IGNITE-13488 - Command to print metric value
>> IGNITE-13426 - Command to print system view content
>> IGNITE-13422 - Parameter to explicitly enable experimental commands
>>
>> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>>
>> New system views:
>>
>> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
>> IGNITE-13408 BinaryMetadata view.
>>
>>> 9 окт. 2020 г., в 04:04, Denis Magda <[hidden email]> написал(а):
>>>
>>> @Alex Plehanov <[hidden email]>,
>>>
>>> If it still makes sense and not too late, you can cherry-pick the commit
>>> with the new docs into the 2.9 branch:
>>>
>> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
>>>
>>> Anyway, it's not crucial as long as we agreed to make an exception for
>> this
>>> release. The docs were already published to the website and assembled
>> from
>>> the master. Once the vote passes, I'll make them visible and traceable
>> from
>>> the website menus.
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
>> [hidden email]>
>>> wrote:
>>>
>>>> Saikat,
>>>>
>>>> The plan sounds good to me! Do you have an idea for the timeline of the
>>>> module releases? Let me know if I can help in any way.
>>>>
>>>> Also, I was looking into the modularization IEP and noticed that OSGI
>>>> module is discussed to be moved to the extensions, but there is no
>>>> corresponding ticket. Should I create one? I will be happy to help with
>>>> moving this module to extensions.
>>>>
>>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]>:
>>>>
>>>>> Hi Alexey,
>>>>>
>>>>> All the modules as planned in IEP-36
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>>>>> are migrated to ignite-extensions repository.
>>>>>
>>>>> We can definitely plan to release ignite-extensions modules.
>>>>>
>>>>> I have a pending PR related to the kafka module and the PR is in review
>>>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
>> has
>>>>> been merged in ignite-extensions repository.
>>>>>
>>>>> My thoughts / action items for the migration efforts and releases were
>>>>> as follows:
>>>>>
>>>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>>>>> 2. Fix the upload nightly packages task for ignite-core to help fix the
>>>>> travis build failure in ignite-extensions repository.
>>>>> 3. Review and update the README.md files for the ignite-extensions
>>>> modules
>>>>> as required.
>>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
>>>>> 5. Release each module separately and share updates.
>>>>>
>>>>> Please let me know if you have feedback.
>>>>>
>>>>> Regards,
>>>>> Saikat
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]>
>> wrote:
>>>>>
>>>>>> It seems that gitbox.apache.org is not available on CI/CD.
>>>>>>
>>>>>> I will try to update VCS to GitHub.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>> Guys,
>>>>>>>
>>>>>>> I've tried to build release candidate using TC [1]. But:
>>>>>>> 1. I can't choose a branch in this TC task (there is no such field).
>>>>>>> 2. On the "default" branch I got an error: Failed to collect changes,
>>>>>>> error: List remote refs failed:
>>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
>>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>>>> connect
>>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
>>>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>>>>>>
>>>>>>> Is there some problem with this TC task? What am I doing wrong?
>>>>>>>
>>>>>>> [1] :
>>>>>>>
>>>>>>
>>>>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>>>>>>
>>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>>>>> [hidden email]>:
>>>>>>>
>>>>>>>> Saikat, Nikolay,
>>>>>>>>
>>>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
>>>>>> Given
>>>>>>>> that these modules will not be available in Ignite 2.9 anymore (will
>>>>>>>> they?), should we also plan to release the extensions?
>>>>>>>>
>>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> [hidden email]
>>>>> :
>>>>>>>>
>>>>>>>>> Igniters,
>>>>>>>>>
>>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
>>>> review
>>>>>>>> it,
>>>>>>>>> please?
>>>>>>>>>
>>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
>>>>>>>>>
>>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>>>> [hidden email]
>>>>>>> :
>>>>>>>>>
>>>>>>>>>> Guys,
>>>>>>>>>>
>>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>>>>> This
>>>>>>>>> bug
>>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>>>>>> and
>>>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
>>>> visa)
>>>>>>>> if
>>>>>>>>>> there are no objections.
>>>>>>>>>>
>>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>>>>>>>>>
>>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>>>> [hidden email]
>>>>>>> :
>>>>>>>>>>
>>>>>>>>>>> Guys,
>>>>>>>>>>>
>>>>>>>>>>> During internal testing, we've found a critical bug with
>>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
>>>> sequentially).
>>>>>>>>> This
>>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>>>>>>>>>> before release. Under investigation now. I'll let you know when
>> we
>>>>>> get
>>>>>>>>>>> something.
>>>>>>>>>>>
>>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
>>>>>>>>>>>
>>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>>> of
>>>>>>>>> the
>>>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>>>> in
>>>>>>>>> later
>>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>>>> next
>>>>>>>>> releases?
>>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> the
>>>>>>>>> factory
>>>>>>>>>>>> closures for certain messages.
>>>>>>>>>>>>
>>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>>>> all
>>>>>>>>>>>> workloads (I didn't get any significant drops during
>>>> benchmarking).
>>>>>>>> So
>>>>>>>>>>>> I prefer the runtime generation in later releases.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>>>>>>>>>>> <[hidden email]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alexey, Andrey, Igniters,
>>>>>>>>>>>>>
>>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>>> of
>>>>>>>>> the
>>>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>>>> in
>>>>>>>>> later
>>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>>>> next
>>>>>>>>> releases?
>>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> the
>>>>>>>>> factory
>>>>>>>>>>>> closures for certain messages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Personally, I would prefer fixing the regression given that we
>>>>>> also
>>>>>>>>>>>> introduced tracing in this release.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>>>>>>> [hidden email]
>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>>>> ed52559eb95
>>>>>>>>> and
>>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
>>>> average
>>>>>> on
>>>>>>>>> our
>>>>>>>>>>>> environment (about the same results we got benchmarking 65c30ec6
>>>> vs
>>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
>>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>> this
>>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>>>> servers, 5
>>>>>>>>> clients
>>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
>>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>>>>>>>>>>> [hidden email]>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
>>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>> you
>>>>>> can
>>>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
>>>>>> won't
>>>>>>>> be
>>>>>>>>>>>> able to spend any time on this project any longer. You can send
>>>>>> your
>>>>>>>>> pull
>>>>>>>>>>>> requests and questions about the documentation to Denis Magda.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Artem
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] :
>>>>>>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>>>>>>>>>>> [hidden email]> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
>>>>>>>>>>>> unfortunately, I
>>>>>>>>>>>>>>>> cannot spot the difference between old and new
>> implementation
>>>>>> in
>>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>>>>>>>>>>> MessageFactoryImpl
>>>>>>>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>>>>>>>> branch
>>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>>>>>>> compatible
>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you check if this change makes any difference
>>>>>>>> performance-wise
>>>>>>>>>>>> in your
>>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code generation
>>>> in
>>>>>>>> the
>>>>>>>>>>>> long
>>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
>> factory
>>>>>>>> with
>>>>>>>>>>>> a switch
>>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
>>>> though,
>>>>>>>>>>>> obviously).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>>>>>>>> [hidden email]
>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello guys,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
>>>> it
>>>>>>>>>>>> reduced the
>>>>>>>>>>>>>>>>> drop, but not completely removed it.
>>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>>>> patch?
>>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>>>> against
>>>>>>>>>>>> 2.8.1
>>>>>>>>>>>>>>>>> release on your environment?
>>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>>>> left,
>>>>>>>>>>>> it's close
>>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
>>>>>>>>>>>> showstopper. Guys,
>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>>>>>>> driver
>>>>>>>>>>>> between 2.8
>>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>>>>>>> should
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>>>>>>>> message
>>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
>>>> if
>>>>>>>>>>>> there is no
>>>>>>>>>>>>>>>>> objection.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>>>>>>> [hidden email]
>>>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>>>>>>> this
>>>>>>>>>>>> issue is
>>>>>>>>>>>>>>>>>> related to the new master key change functionality which
>>>>>>>>>>>> haven't been
>>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> release branch.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>>>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello, Igniters.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>>>>>>> [1]
>>>>>>>>>>>> into the
>>>>>>>>>>>>>>>>> 2.9
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>> fields
>>>>>>>>> in
>>>>>>>>>>>> wrong
>>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>>>>>>>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>>>>>>>>>>> anything out of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>>>>>>>>>>> right now),
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>>>>>>>> the
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>>>>>>>> me
>>>>>>>>>>>> explore
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> code a bit.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>>>>>>> the
>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>>>>>>>>>>> virtual call
>>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
>>>>>>>> amount
>>>>>>>>>>>> of other
>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>> happening during the message processing.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>>>>>>>> results.
>>>>>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>>>>>>>> between
>>>>>>>>>>>> e6a7f93
>>>>>>>>>>>>>>>>>> (first
>>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>>>>>>>>>>> 6592dfa5 (last
>>>>>>>>>>>>>>>>>> commit in
>>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>>>>>>>>>>> commits.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>>>>>>> a
>>>>>>>>>>>> drop
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>>>>>>>>>>> reverted
>>>>>>>>>>>>>>>>>> IGNITE-13060
>>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>>>>>>>>>>> related to
>>>>>>>>>>>>>>>>>> drop
>>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>>>>>>>>>>> problem, and
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>>>>>>> leave
>>>>>>>>>>>> it as is?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>>>>>>>>>>> refactoring)?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>>>>>>>>>>> incorrect fix
>>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>>>>>>>>>>> [1], so it
>>>>>>>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>>>>>>> work
>>>>>>>>>>>> with fix
>>>>>>>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>>>>>>>>>>> versions.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --AG
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>>>>>>>>>> IGNITE-12568
>>>>>>>>>>>>>>>>>> (reverted
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>>>>>>>>>>> paths, to
>>>>>>>>>>>>>>>>> trace
>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>>>>>>>> switch/case
>>>>>>>>>>>> block was
>>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>>>>>>>>>>> message factory
>>>>>>>>>>>>>>>>>> is on
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>>>>>>> impact
>>>>>>>>>>>> on total
>>>>>>>>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>>>>>>>>>>> microbenchmarks,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>>>>>>>>>>> about 30-35%
>>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>>>>>>>> can
>>>>>>>>>>>>>>>>> effectively
>>>>>>>>>>>>>>>>>>>>>>>> inline
>>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>>>>>>> suppliers
>>>>>>>>>>>> relatively
>>>>>>>>>>>>>>>>>> heavy
>>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>>>>>>>>>>> rewrite the
>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>>>>>>>> interface
>>>>>>>>>>>> (to
>>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>>>>>>>>>>> but it gives
>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>>>>>>> looks
>>>>>>>>>>>> ugly
>>>>>>>>>>>>>>>>>> (lambdas
>>>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>>>>>>>>>>> optimize the
>>>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>>>>>>>>>>> Andrey Gura,
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>>>>>>>> about
>>>>>>>>>>>>>>>>>> optimization?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>>>>>>>>>>> some metrics
>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>>>>>>>>>>> factory
>>>>>>>>>>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>>>>>>>>>>> already released
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>>>>>>>>>>> only present
>>>>>>>>>>>>>>>>>> in Ignite
>>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>>>>>>>>>>> metrics
>>>>>>>>>>>>>>>>>> created for
>>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>>>>>>>>>>> the release
>>>>>>>>>>>>>>>>>> and deal
>>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>>

[Alexey Plekhanov]

Guys,

I've found 3 more tickets which were targeted to 2.9 but merged only to the
master branch ([1], [2], [3]).
Do we need these tickets in 2.9? Are they critical?

[1]: https://issues.apache.org/jira/browse/IGNITE-12350
[2]: https://issues.apache.org/jira/browse/IGNITE-13376
[3]: https://issues.apache.org/jira/browse/IGNITE-13280

вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov <[hidden email]>:

> Igniters,
>
> IGNITE-13553 fixed and cherry-picked to 2.9.
> We can continue with the release.
>
> > 10 окт. 2020 г., в 00:00, Denis Magda <[hidden email]> написал(а):
> >
> > Nikolay,
> >
> > re: the minor improvements. I'm not against of including those if we can
> > prepare the docs before the vote starts. Presently, the docs are "frozen"
> > for the 2.9 release, but I can scratch some time and take part in the
> docs
> > review the next week.
> >
> > -
> > Denis
> >
> >
> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <[hidden email]>
> wrote:
> >
> >> Hello, Igniters.
> >>
> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> >> stops in case wrong datatype put in indexed field»
> >> Can someone, please, help me with the review?
> >>
> >> [1] https://github.com/apache/ignite/pull/8330
> >> [2] https://issues.apache.org/jira/browse/IGNITE-13553
> >>
> >> ==================
> >>
> >> I propose to include several minor tickets to the scope of the 2.9 that
> >> increase Ignite User Experience
> >>
> >> CMD tools improvements:
> >>
> >> IGNITE-13488 - Command to print metric value
> >> IGNITE-13426 - Command to print system view content
> >> IGNITE-13422 - Parameter to explicitly enable experimental commands
> >>
> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
> >>
> >> New system views:
> >>
> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> >> IGNITE-13408 BinaryMetadata view.
> >>
> >>> 9 окт. 2020 г., в 04:04, Denis Magda <[hidden email]> написал(а):
> >>>
> >>> @Alex Plehanov <[hidden email]>,
> >>>
> >>> If it still makes sense and not too late, you can cherry-pick the
> commit
> >>> with the new docs into the 2.9 branch:
> >>>
> >>
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> >>>
> >>> Anyway, it's not crucial as long as we agreed to make an exception for
> >> this
> >>> release. The docs were already published to the website and assembled
> >> from
> >>> the master. Once the vote passes, I'll make them visible and traceable
> >> from
> >>> the website menus.
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> >> [hidden email]>
> >>> wrote:
> >>>
> >>>> Saikat,
> >>>>
> >>>> The plan sounds good to me! Do you have an idea for the timeline of
> the
> >>>> module releases? Let me know if I can help in any way.
> >>>>
> >>>> Also, I was looking into the modularization IEP and noticed that OSGI
> >>>> module is discussed to be moved to the extensions, but there is no
> >>>> corresponding ticket. Should I create one? I will be happy to help
> with
> >>>> moving this module to extensions.
> >>>>
> >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <[hidden email]
> >:
> >>>>
> >>>>> Hi Alexey,
> >>>>>
> >>>>> All the modules as planned in IEP-36
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> >>>>> are migrated to ignite-extensions repository.
> >>>>>
> >>>>> We can definitely plan to release ignite-extensions modules.
> >>>>>
> >>>>> I have a pending PR related to the kafka module and the PR is in
> review
> >>>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
> >> has
> >>>>> been merged in ignite-extensions repository.
> >>>>>
> >>>>> My thoughts / action items for the migration efforts and releases
> were
> >>>>> as follows:
> >>>>>
> >>>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> >>>>> 2. Fix the upload nightly packages task for ignite-core to help fix
> the
> >>>>> travis build failure in ignite-extensions repository.
> >>>>> 3. Review and update the README.md files for the ignite-extensions
> >>>> modules
> >>>>> as required.
> >>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
> >>>>> 5. Release each module separately and share updates.
> >>>>>
> >>>>> Please let me know if you have feedback.
> >>>>>
> >>>>> Regards,
> >>>>> Saikat
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <[hidden email]>
> >> wrote:
> >>>>>
> >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
> >>>>>>
> >>>>>> I will try to update VCS to GitHub.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <[hidden email]>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Guys,
> >>>>>>>
> >>>>>>> I've tried to build release candidate using TC [1]. But:
> >>>>>>> 1. I can't choose a branch in this TC task (there is no such
> field).
> >>>>>>> 2. On the "default" branch I got an error: Failed to collect
> changes,
> >>>>>>> error: List remote refs failed:
> >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> >>>> connect
> >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> >>>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >>>>>>>
> >>>>>>> Is there some problem with this TC task? What am I doing wrong?
> >>>>>>>
> >>>>>>> [1] :
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >>>>>>>
> >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >>>>>> [hidden email]>:
> >>>>>>>
> >>>>>>>> Saikat, Nikolay,
> >>>>>>>>
> >>>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
> >>>>>> Given
> >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
> (will
> >>>>>>>> they?), should we also plan to release the extensions?
> >>>>>>>>
> >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> >> [hidden email]
> >>>>> :
> >>>>>>>>
> >>>>>>>>> Igniters,
> >>>>>>>>>
> >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
> >>>> review
> >>>>>>>> it,
> >>>>>>>>> please?
> >>>>>>>>>
> >>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
> >>>>>>>>>
> >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> >>>> [hidden email]
> >>>>>>> :
> >>>>>>>>>
> >>>>>>>>>> Guys,
> >>>>>>>>>>
> >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
> bug.
> >>>>>> This
> >>>>>>>>> bug
> >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
> privately
> >>>>>> and
> >>>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
> >>>> visa)
> >>>>>>>> if
> >>>>>>>>>> there are no objections.
> >>>>>>>>>>
> >>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >>>>>>>>>>
> >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> >>>> [hidden email]
> >>>>>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Guys,
> >>>>>>>>>>>
> >>>>>>>>>>> During internal testing, we've found a critical bug with
> >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
> >>>> sequentially).
> >>>>>>>>> This
> >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >>>>>>>>>>> before release. Under investigation now. I'll let you know when
> >> we
> >>>>>> get
> >>>>>>>>>>> something.
> >>>>>>>>>>>
> >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <[hidden email]>:
> >>>>>>>>>>>
> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> version
> >>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> generation
> >>>> in
> >>>>>>>>> later
> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
> >>>> next
> >>>>>>>>> releases?
> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> fairly
> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
> >> the
> >>>>>>>>> factory
> >>>>>>>>>>>> closures for certain messages.
> >>>>>>>>>>>>
> >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
> for
> >>>> all
> >>>>>>>>>>>> workloads (I didn't get any significant drops during
> >>>> benchmarking).
> >>>>>>>> So
> >>>>>>>>>>>> I prefer the runtime generation in later releases.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >>>>>>>>>>>> <[hidden email]> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alexey, Andrey, Igniters,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> version
> >>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> generation
> >>>> in
> >>>>>>>>> later
> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
> >>>> next
> >>>>>>>>> releases?
> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> fairly
> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
> >> the
> >>>>>>>>> factory
> >>>>>>>>>>>> closures for certain messages.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Personally, I would prefer fixing the regression given that
> we
> >>>>>> also
> >>>>>>>>>>>> introduced tracing in this release.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >>>>>>>> [hidden email]
> >>>>>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> >>>> ed52559eb95
> >>>>>>>>> and
> >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> >>>> average
> >>>>>> on
> >>>>>>>>> our
> >>>>>>>>>>>> environment (about the same results we got benchmarking
> 65c30ec6
> >>>> vs
> >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
> >> this
> >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> >>>> servers, 5
> >>>>>>>>> clients
> >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >>>>>>>>>>>> [hidden email]>:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
> >> you
> >>>>>> can
> >>>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
> >>>>>> won't
> >>>>>>>> be
> >>>>>>>>>>>> able to spend any time on this project any longer. You can
> send
> >>>>>> your
> >>>>>>>>> pull
> >>>>>>>>>>>> requests and questions about the documentation to Denis Magda.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Artem
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1] :
> >>>>>>>>>>>>
> >>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>>>>>>>>>>> [hidden email]> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
> >>>>>>>>>>>> unfortunately, I
> >>>>>>>>>>>>>>>> cannot spot the difference between old and new
> >> implementation
> >>>>>> in
> >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >>>>>>>>>>>> MessageFactoryImpl
> >>>>>>>>>>>>>>>> with the old switch statement to the
> ignite-2.9-revert-12568
> >>>>>>>>> branch
> >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >>>>>>>> compatible
> >>>>>>>>>>>> with the
> >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can you check if this change makes any difference
> >>>>>>>> performance-wise
> >>>>>>>>>>>> in your
> >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
> generation
> >>>> in
> >>>>>>>> the
> >>>>>>>>>>>> long
> >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
> >> factory
> >>>>>>>> with
> >>>>>>>>>>>> a switch
> >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> >>>> though,
> >>>>>>>>>>>> obviously).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >>>>>>>>> [hidden email]
> >>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello guys,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
> [1]),
> >>>> it
> >>>>>>>>>>>> reduced the
> >>>>>>>>>>>>>>>>> drop, but not completely removed it.
> >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> >>>> patch?
> >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> >>>> against
> >>>>>>>>>>>> 2.8.1
> >>>>>>>>>>>>>>>>> release on your environment?
> >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> >>>> left,
> >>>>>>>>>>>> it's close
> >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> >>>>>>>>>>>> showstopper. Guys,
> >>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >>>>>>>> driver
> >>>>>>>>>>>> between 2.8
> >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >>>>>>>> should
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
> (NIO
> >>>>>>>>> message
> >>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick
> it
> >>>> if
> >>>>>>>>>>>> there is no
> >>>>>>>>>>>>>>>>> objection.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >>>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >>>>>>>> [hidden email]
> >>>>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >>>>>>>> this
> >>>>>>>>>>>> issue is
> >>>>>>>>>>>>>>>>>> related to the new master key change functionality which
> >>>>>>>>>>>> haven't been
> >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
> commit
> >>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> release branch.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >>>>>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello, Igniters.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >>>>>>>> [1]
> >>>>>>>>>>>> into the
> >>>>>>>>>>>>>>>>> 2.9
> >>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> >> fields
> >>>>>>>>> in
> >>>>>>>>>>>> wrong
> >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >>>>>>>>>>>> anything out of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
> ideas
> >>>>>>>>>>>> right now),
> >>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
> the
> >>>>>>>>>>>> switch
> >>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
> inlining
> >>>>>>>>> the
> >>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>> messages
> >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
> Let
> >>>>>>>>> me
> >>>>>>>>>>>> explore
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> code a bit.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >>>>>>>> the
> >>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
> single
> >>>>>>>>>>>> virtual call
> >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> >>>>>>>> amount
> >>>>>>>>>>>> of other
> >>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>> happening during the message processing.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >>>>>>>>> results.
> >>>>>>>>>>>>>>>>> Actually,
> >>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >>>>>>>>> between
> >>>>>>>>>>>> e6a7f93
> >>>>>>>>>>>>>>>>>> (first
> >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >>>>>>>>>>>> 6592dfa5 (last
> >>>>>>>>>>>>>>>>>> commit in
> >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >>>>>>>>>>>> commits.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
> for
> >>>>>>>> a
> >>>>>>>>>>>> drop
> >>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >>>>>>>>>>>> reverted
> >>>>>>>>>>>>>>>>>> IGNITE-13060
> >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
> not
> >>>>>>>>>>>> related to
> >>>>>>>>>>>>>>>>>> drop
> >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >>>>>>>>>>>> problem, and
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >>>>>>>> leave
> >>>>>>>>>>>> it as is?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >>>>>>>>>>>> refactoring)?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has
> an
> >>>>>>>>>>>> incorrect fix
> >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
> branch
> >>>>>>>>>>>> [1], so it
> >>>>>>>>>>>>>>>>>> cannot be
> >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >>>>>>>> work
> >>>>>>>>>>>> with fix
> >>>>>>>>>>>>>>>>>> versions
> >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >>>>>>>>>>>> versions.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --AG
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>>>>>>>>>>>>>>> [hidden email]
> >>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Guys,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>>>>>>>>>>> IGNITE-12568
> >>>>>>>>>>>>>>>>>> (reverted
> >>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
> hot
> >>>>>>>>>>>> paths, to
> >>>>>>>>>>>>>>>>> trace
> >>>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >>>>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >>>>>>>>> switch/case
> >>>>>>>>>>>> block was
> >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >>>>>>>>>>>> message factory
> >>>>>>>>>>>>>>>>>> is on
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >>>>>>>> impact
> >>>>>>>>>>>> on total
> >>>>>>>>>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >>>>>>>>>>>> microbenchmarks,
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >>>>>>>>>>>> about 30-35%
> >>>>>>>>>>>>>>>>>> faster
> >>>>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
> switch/case
> >>>>>>>>> can
> >>>>>>>>>>>>>>>>> effectively
> >>>>>>>>>>>>>>>>>>>>>>>> inline
> >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >>>>>>>> suppliers
> >>>>>>>>>>>> relatively
> >>>>>>>>>>>>>>>>>> heavy
> >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >>>>>>>>>>>> rewrite the
> >>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >>>>>>>>> interface
> >>>>>>>>>>>> (to
> >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
> "invokevirtual"),
> >>>>>>>>>>>> but it gives
> >>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >>>>>>>> looks
> >>>>>>>>>>>> ugly
> >>>>>>>>>>>>>>>>>> (lambdas
> >>>>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >>>>>>>>>>>> optimize the
> >>>>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >>>>>>>>>>>> Andrey Gura,
> >>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >>>>>>>>> about
> >>>>>>>>>>>>>>>>>> optimization?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
> are
> >>>>>>>>>>>> some metrics
> >>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
> message
> >>>>>>>>>>>> factory
> >>>>>>>>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >>>>>>>>>>>> already released
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory)
> is
> >>>>>>>>>>>> only present
> >>>>>>>>>>>>>>>>>> in Ignite
> >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
> new
> >>>>>>>>>>>> metrics
> >>>>>>>>>>>>>>>>>> created for
> >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
> unblock
> >>>>>>>>>>>> the release
> >>>>>>>>>>>>>>>>>> and deal
> >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>
> >>
>
>

[Zhenya Stanilovsky]

without [2] and [3] we obtain unexpected fields in index creation and as a consequence buggy sql execution and pla of course.


 

>Guys,
>
>I've found 3 more tickets which were targeted to 2.9 but merged only to the
>master branch ([1], [2], [3]).
>Do we need these tickets in 2.9? Are they critical?
>
>[1]:  https://issues.apache.org/jira/browse/IGNITE-12350
>[2]:  https://issues.apache.org/jira/browse/IGNITE-13376
>[3]:  https://issues.apache.org/jira/browse/IGNITE-13280
>
>вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov < [hidden email] >:
> 
>> Igniters,
>>
>> IGNITE-13553 fixed and cherry-picked to 2.9.
>> We can continue with the release.
>>
>> > 10 окт. 2020 г., в 00:00, Denis Magda < [hidden email] > написал(а):
>> >
>> > Nikolay,
>> >
>> > re: the minor improvements. I'm not against of including those if we can
>> > prepare the docs before the vote starts. Presently, the docs are "frozen"
>> > for the 2.9 release, but I can scratch some time and take part in the
>> docs
>> > review the next week.
>> >
>> > -
>> > Denis
>> >
>> >
>> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov < [hidden email] >
>> wrote:
>> >
>> >> Hello, Igniters.
>> >>
>> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
>> >> stops in case wrong datatype put in indexed field»
>> >> Can someone, please, help me with the review?
>> >>
>> >> [1]  https://github.com/apache/ignite/pull/8330
>> >> [2]  https://issues.apache.org/jira/browse/IGNITE-13553
>> >>
>> >> ==================
>> >>
>> >> I propose to include several minor tickets to the scope of the 2.9 that
>> >> increase Ignite User Experience
>> >>
>> >> CMD tools improvements:
>> >>
>> >> IGNITE-13488 - Command to print metric value
>> >> IGNITE-13426 - Command to print system view content
>> >> IGNITE-13422 - Parameter to explicitly enable experimental commands
>> >>
>> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>> >>
>> >> New system views:
>> >>
>> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
>> >> IGNITE-13408 BinaryMetadata view.
>> >>
>> >>> 9 окт. 2020 г., в 04:04, Denis Magda < [hidden email] > написал(а):
>> >>>
>> >>> @Alex Plehanov < [hidden email] >,
>> >>>
>> >>> If it still makes sense and not too late, you can cherry-pick the
>> commit
>> >>> with the new docs into the 2.9 branch:
>> >>>
>> >>
>>  https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
>> >>>
>> >>> Anyway, it's not crucial as long as we agreed to make an exception for
>> >> this
>> >>> release. The docs were already published to the website and assembled
>> >> from
>> >>> the master. Once the vote passes, I'll make them visible and traceable
>> >> from
>> >>> the website menus.
>> >>>
>> >>> -
>> >>> Denis
>> >>>
>> >>>
>> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
>> >>  [hidden email] >
>> >>> wrote:
>> >>>
>> >>>> Saikat,
>> >>>>
>> >>>> The plan sounds good to me! Do you have an idea for the timeline of
>> the
>> >>>> module releases? Let me know if I can help in any way.
>> >>>>
>> >>>> Also, I was looking into the modularization IEP and noticed that OSGI
>> >>>> module is discussed to be moved to the extensions, but there is no
>> >>>> corresponding ticket. Should I create one? I will be happy to help
>> with
>> >>>> moving this module to extensions.
>> >>>>
>> >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra < [hidden email]
>> >:
>> >>>>
>> >>>>> Hi Alexey,
>> >>>>>
>> >>>>> All the modules as planned in IEP-36
>> >>>>>
>> >>>>
>> >>
>>  https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> >>>>> are migrated to ignite-extensions repository.
>> >>>>>
>> >>>>> We can definitely plan to release ignite-extensions modules.
>> >>>>>
>> >>>>> I have a pending PR related to the kafka module and the PR is in
>> review
>> >>>>>  https://github.com/apache/ignite/pull/8222 but its corresponding PR
>> >> has
>> >>>>> been merged in ignite-extensions repository.
>> >>>>>
>> >>>>> My thoughts / action items for the migration efforts and releases
>> were
>> >>>>> as follows:
>> >>>>>
>> >>>>> 1. Merge the changes for  https://github.com/apache/ignite/pull/8222
>> >>>>> 2. Fix the upload nightly packages task for ignite-core to help fix
>> the
>> >>>>> travis build failure in ignite-extensions repository.
>> >>>>> 3. Review and update the README.md files for the ignite-extensions
>> >>>> modules
>> >>>>> as required.
>> >>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
>> >>>>> 5. Release each module separately and share updates.
>> >>>>>
>> >>>>> Please let me know if you have feedback.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Saikat
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov < [hidden email] >
>> >> wrote:
>> >>>>>
>> >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
>> >>>>>>
>> >>>>>> I will try to update VCS to GitHub.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov < [hidden email] >
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Guys,
>> >>>>>>>
>> >>>>>>> I've tried to build release candidate using TC [1]. But:
>> >>>>>>> 1. I can't choose a branch in this TC task (there is no such
>> field).
>> >>>>>>> 2. On the "default" branch I got an error: Failed to collect
>> changes,
>> >>>>>>> error: List remote refs failed:
>> >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
>> >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> >>>> connect
>> >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
>> >>>>>>>  https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master "}
>> >>>>>>>
>> >>>>>>> Is there some problem with this TC task? What am I doing wrong?
>> >>>>>>>
>> >>>>>>> [1] :
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>>  https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >>>>>>>
>> >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> >>>>>>  [hidden email] >:
>> >>>>>>>
>> >>>>>>>> Saikat, Nikolay,
>> >>>>>>>>
>> >>>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
>> >>>>>> Given
>> >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
>> (will
>> >>>>>>>> they?), should we also plan to release the extensions?
>> >>>>>>>>
>> >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> >>  [hidden email]
>> >>>>> :
>> >>>>>>>>
>> >>>>>>>>> Igniters,
>> >>>>>>>>>
>> >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
>> >>>> review
>> >>>>>>>> it,
>> >>>>>>>>> please?
>> >>>>>>>>>
>> >>>>>>>>> [1]:  https://github.com/apache/ignite/pull/8273
>> >>>>>>>>>
>> >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>> >>>>  [hidden email]
>> >>>>>>> :
>> >>>>>>>>>
>> >>>>>>>>>> Guys,
>> >>>>>>>>>>
>> >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
>> bug.
>> >>>>>> This
>> >>>>>>>>> bug
>> >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
>> privately
>> >>>>>> and
>> >>>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
>> >>>> visa)
>> >>>>>>>> if
>> >>>>>>>>>> there are no objections.
>> >>>>>>>>>>
>> >>>>>>>>>> [1]:  https://issues.apache.org/jira/browse/IGNITE-13465
>> >>>>>>>>>> [2]:  https://issues.apache.org/jira/browse/IGNITE-13134
>> >>>>>>>>>>
>> >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>> >>>>  [hidden email]
>> >>>>>>> :
>> >>>>>>>>>>
>> >>>>>>>>>>> Guys,
>> >>>>>>>>>>>
>> >>>>>>>>>>> During internal testing, we've found a critical bug with
>> >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
>> >>>> sequentially).
>> >>>>>>>>> This
>> >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>> >>>>>>>>>>> before release. Under investigation now. I'll let you know when
>> >> we
>> >>>>>> get
>> >>>>>>>>>>> something.
>> >>>>>>>>>>>
>> >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura < [hidden email] >:
>> >>>>>>>>>>>
>> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version
>> >>>> of
>> >>>>>>>>> the
>> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
>> generation
>> >>>> in
>> >>>>>>>>> later
>> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> >>>> next
>> >>>>>>>>> releases?
>> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
>> fairly
>> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> >> the
>> >>>>>>>>> factory
>> >>>>>>>>>>>> closures for certain messages.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
>> for
>> >>>> all
>> >>>>>>>>>>>> workloads (I didn't get any significant drops during
>> >>>> benchmarking).
>> >>>>>>>> So
>> >>>>>>>>>>>> I prefer the runtime generation in later releases.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> >>>>>>>>>>>> < [hidden email] > wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Alexey, Andrey, Igniters,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version
>> >>>> of
>> >>>>>>>>> the
>> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
>> generation
>> >>>> in
>> >>>>>>>>> later
>> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> >>>> next
>> >>>>>>>>> releases?
>> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
>> fairly
>> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> >> the
>> >>>>>>>>> factory
>> >>>>>>>>>>>> closures for certain messages.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Personally, I would prefer fixing the regression given that
>> we
>> >>>>>> also
>> >>>>>>>>>>>> introduced tracing in this release.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> >>>>>>>>  [hidden email]
>> >>>>>>>>>> :
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>> >>>> ed52559eb95
>> >>>>>>>>> and
>> >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
>> >>>> average
>> >>>>>> on
>> >>>>>>>>> our
>> >>>>>>>>>>>> environment (about the same results we got benchmarking
>> 65c30ec6
>> >>>> vs
>> >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
>> >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>> >> this
>> >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>> >>>> servers, 5
>> >>>>>>>>> clients
>> >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
>> >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> >>>>>>>>>>>>  [hidden email] >:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi Everyone,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
>> >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>> >> you
>> >>>>>> can
>> >>>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
>> >>>>>> won't
>> >>>>>>>> be
>> >>>>>>>>>>>> able to spend any time on this project any longer. You can
>> send
>> >>>>>> your
>> >>>>>>>>> pull
>> >>>>>>>>>>>> requests and questions about the documentation to Denis Magda.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> -Artem
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> [1] :
>> >>>>>>>>>>>>
>> >>>>  https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> >>>>>>>>>>>>  [hidden email] > wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
>> >>>>>>>>>>>> unfortunately, I
>> >>>>>>>>>>>>>>>> cannot spot the difference between old and new
>> >> implementation
>> >>>>>> in
>> >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>> >>>>>>>>>>>> MessageFactoryImpl
>> >>>>>>>>>>>>>>>> with the old switch statement to the
>> ignite-2.9-revert-12568
>> >>>>>>>>> branch
>> >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> >>>>>>>> compatible
>> >>>>>>>>>>>> with the
>> >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Can you check if this change makes any difference
>> >>>>>>>> performance-wise
>> >>>>>>>>>>>> in your
>> >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
>> generation
>> >>>> in
>> >>>>>>>> the
>> >>>>>>>>>>>> long
>> >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
>> >> factory
>> >>>>>>>> with
>> >>>>>>>>>>>> a switch
>> >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
>> >>>> though,
>> >>>>>>>>>>>> obviously).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>> >>>>>>>>>  [hidden email]
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Hello guys,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
>> [1]),
>> >>>> it
>> >>>>>>>>>>>> reduced the
>> >>>>>>>>>>>>>>>>> drop, but not completely removed it.
>> >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>> >>>> patch?
>> >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>> >>>> against
>> >>>>>>>>>>>> 2.8.1
>> >>>>>>>>>>>>>>>>> release on your environment?
>> >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>> >>>> left,
>> >>>>>>>>>>>> it's close
>> >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
>> >>>>>>>>>>>> showstopper. Guys,
>> >>>>>>>>>>>>>>>>> WDYT?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> >>>>>>>> driver
>> >>>>>>>>>>>> between 2.8
>> >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> >>>>>>>> should
>> >>>>>>>>>>>> be
>> >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>> >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
>> (NIO
>> >>>>>>>>> message
>> >>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick
>> it
>> >>>> if
>> >>>>>>>>>>>> there is no
>> >>>>>>>>>>>>>>>>> objection.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> [1]  https://issues.apache.org/jira/browse/IGNITE-13411
>> >>>>>>>>>>>>>>>>> [2]  https://github.com/apache/ignite/pull/8223
>> >>>>>>>>>>>>>>>>> [3]  https://issues.apache.org/jira/browse/IGNITE-13414
>> >>>>>>>>>>>>>>>>> [4]  https://issues.apache.org/jira/browse/IGNITE-13361
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> >>>>>>>>  [hidden email]
>> >>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> >>>>>>>> this
>> >>>>>>>>>>>> issue is
>> >>>>>>>>>>>>>>>>>> related to the new master key change functionality which
>> >>>>>>>>>>>> haven't been
>> >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
>> commit
>> >>>>>>>> to
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> release branch.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> [1]  https://issues.apache.org/jira/browse/IGNITE-13390
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> >>>>>>>>>>>>  [hidden email] >
>> >>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Hello, Igniters.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> >>>>>>>> [1]
>> >>>>>>>>>>>> into the
>> >>>>>>>>>>>>>>>>> 2.9
>> >>>>>>>>>>>>>>>>>> release
>> >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>> >> fields
>> >>>>>>>>> in
>> >>>>>>>>>>>> wrong
>> >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> [1]  https://issues.apache.org/jira/browse/IGNITE-12809
>> >>>>>>>>>>>>>>>>>>> [2]
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>>  https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >>>>>>>>>>>>>>>>>  [hidden email] >
>> >>>>>>>>>>>>>>>>>> написал(а):
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>> >>>>>>>>>>>> anything out of
>> >>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
>> ideas
>> >>>>>>>>>>>> right now),
>> >>>>>>>>>>>>>>>>> so
>> >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
>> the
>> >>>>>>>>>>>> switch
>> >>>>>>>>>>>>>>>>> approach
>> >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
>> inlining
>> >>>>>>>>> the
>> >>>>>>>>>>>> Ignite
>> >>>>>>>>>>>>>>>>>> messages
>> >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
>> Let
>> >>>>>>>>> me
>> >>>>>>>>>>>> explore
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> code a bit.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> >>>>>>>> the
>> >>>>>>>>>>>>>>>>> performance.
>> >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
>> single
>> >>>>>>>>>>>> virtual call
>> >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
>> >>>>>>>> amount
>> >>>>>>>>>>>> of other
>> >>>>>>>>>>>>>>>>>> work
>> >>>>>>>>>>>>>>>>>>>> happening during the message processing.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> >>>>>>>>>>>>  [hidden email]
>> >>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>> >>>>>>>>> results.
>> >>>>>>>>>>>>>>>>> Actually,
>> >>>>>>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>> >>>>>>>>> between
>> >>>>>>>>>>>> e6a7f93
>> >>>>>>>>>>>>>>>>>> (first
>> >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>> >>>>>>>>>>>> 6592dfa5 (last
>> >>>>>>>>>>>>>>>>>> commit in
>> >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>> >>>>>>>>>>>> commits.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
>> for
>> >>>>>>>> a
>> >>>>>>>>>>>> drop
>> >>>>>>>>>>>>>>>>> between
>> >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> >>>>>>>>>>>> reverted
>> >>>>>>>>>>>>>>>>>> IGNITE-13060
>> >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
>> not
>> >>>>>>>>>>>> related to
>> >>>>>>>>>>>>>>>>>> drop
>> >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>> >>>>>>>>>>>> problem, and
>> >>>>>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>>>> can
>> >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> >>>>>>>> leave
>> >>>>>>>>>>>> it as is?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>> >>>>>>>>>>>> refactoring)?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >>>>>>>>>>>>>>>>>>  [hidden email]
>> >>>>>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has
>> an
>> >>>>>>>>>>>> incorrect fix
>> >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
>> branch
>> >>>>>>>>>>>> [1], so it
>> >>>>>>>>>>>>>>>>>> cannot be
>> >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> >>>>>>>> work
>> >>>>>>>>>>>> with fix
>> >>>>>>>>>>>>>>>>>> versions
>> >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>> >>>>>>>>>>>> versions.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> --AG
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> [1]
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>>  https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >>>>>>>>>>>>>>>>>>  [hidden email]
>> >>>>>>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >>>>>>>>>>>>>>>>>  [hidden email]
>> >>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Guys,
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> >>>>>>>>>>>> IGNITE-12568
>> >>>>>>>>>>>>>>>>>> (reverted
>> >>>>>>>>>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
>> hot
>> >>>>>>>>>>>> paths, to
>> >>>>>>>>>>>>>>>>> trace
>> >>>>>>>>>>>>>>>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>> >>>>>>>>> here.
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>> >>>>>>>>> switch/case
>> >>>>>>>>>>>> block was
>> >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>> >>>>>>>>>>>> message factory
>> >>>>>>>>>>>>>>>>>> is on
>> >>>>>>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> >>>>>>>> impact
>> >>>>>>>>>>>> on total
>> >>>>>>>>>>>>>>>>>>>>>>>> performance.
>> >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>> >>>>>>>>>>>> microbenchmarks,
>> >>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>> found
>> >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>> >>>>>>>>>>>> about 30-35%
>> >>>>>>>>>>>>>>>>>> faster
>> >>>>>>>>>>>>>>>>>>>>>>>> than
>> >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
>> switch/case
>> >>>>>>>>> can
>> >>>>>>>>>>>>>>>>> effectively
>> >>>>>>>>>>>>>>>>>>>>>>>> inline
>> >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> >>>>>>>> suppliers
>> >>>>>>>>>>>> relatively
>> >>>>>>>>>>>>>>>>>> heavy
>> >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>> >>>>>>>>>>>> rewrite the
>> >>>>>>>>>>>>>>>>> code
>> >>>>>>>>>>>>>>>>>>>>>>>> using
>> >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>> >>>>>>>>> interface
>> >>>>>>>>>>>> (to
>> >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
>> "invokevirtual"),
>> >>>>>>>>>>>> but it gives
>> >>>>>>>>>>>>>>>>>> back
>> >>>>>>>>>>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> >>>>>>>> looks
>> >>>>>>>>>>>> ugly
>> >>>>>>>>>>>>>>>>>> (lambdas
>> >>>>>>>>>>>>>>>>>>>>>>>> can't
>> >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>> >>>>>>>>>>>> optimize the
>> >>>>>>>>>>>>>>>>>> current
>> >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>> >>>>>>>>>>>> Andrey Gura,
>> >>>>>>>>>>>>>>>>> as
>> >>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>> >>>>>>>>> about
>> >>>>>>>>>>>>>>>>>> optimization?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
>> are
>> >>>>>>>>>>>> some metrics
>> >>>>>>>>>>>>>>>>>>>>>>>> already
>> >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
>> message
>> >>>>>>>>>>>> factory
>> >>>>>>>>>>>>>>>>>>>>>>>> implementation
>> >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>> >>>>>>>>>>>> already released
>> >>>>>>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory)
>> is
>> >>>>>>>>>>>> only present
>> >>>>>>>>>>>>>>>>>> in Ignite
>> >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
>> new
>> >>>>>>>>>>>> metrics
>> >>>>>>>>>>>>>>>>>> created for
>> >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
>> unblock
>> >>>>>>>>>>>> the release
>> >>>>>>>>>>>>>>>>>> and deal
>> >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>
>> >>
>>
>>

 

[V.Pyatkov]

This patch [1] interesting only a deployment where clients often
reconnecting (stopping/starting) to cluster.
In these cases Coordinator node could fail by OOM, even the cluster did not
use MVCC caches.

On Mon, Oct 12, 2020 at 1:41 PM Zhenya Stanilovsky
<[hidden email]> wrote:

>
>
> without [2] and [3] we obtain unexpected fields in index creation and as a
> consequence buggy sql execution and pla of course.
>
>
>
> >Guys,
> >
> >I've found 3 more tickets which were targeted to 2.9 but merged only to
> the
> >master branch ([1], [2], [3]).
> >Do we need these tickets in 2.9? Are they critical?
> >
> >[1]:  https://issues.apache.org/jira/browse/IGNITE-12350
> >[2]:  https://issues.apache.org/jira/browse/IGNITE-13376
> >[3]:  https://issues.apache.org/jira/browse/IGNITE-13280
> >
> >вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov < [hidden email] >:
> >
> >> Igniters,
> >>
> >> IGNITE-13553 fixed and cherry-picked to 2.9.
> >> We can continue with the release.
> >>
> >> > 10 окт. 2020 г., в 00:00, Denis Magda < [hidden email] >
> написал(а):
> >> >
> >> > Nikolay,
> >> >
> >> > re: the minor improvements. I'm not against of including those if we
> can
> >> > prepare the docs before the vote starts. Presently, the docs are
> "frozen"
> >> > for the 2.9 release, but I can scratch some time and take part in the
> >> docs
> >> > review the next week.
> >> >
> >> > -
> >> > Denis
> >> >
> >> >
> >> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov < [hidden email]
> >
> >> wrote:
> >> >
> >> >> Hello, Igniters.
> >> >>
> >> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> >> >> stops in case wrong datatype put in indexed field»
> >> >> Can someone, please, help me with the review?
> >> >>
> >> >> [1]  https://github.com/apache/ignite/pull/8330
> >> >> [2]  https://issues.apache.org/jira/browse/IGNITE-13553
> >> >>
> >> >> ==================
> >> >>
> >> >> I propose to include several minor tickets to the scope of the 2.9
> that
> >> >> increase Ignite User Experience
> >> >>
> >> >> CMD tools improvements:
> >> >>
> >> >> IGNITE-13488 - Command to print metric value
> >> >> IGNITE-13426 - Command to print system view content
> >> >> IGNITE-13422 - Parameter to explicitly enable experimental commands
> >> >>
> >> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
> >> >>
> >> >> New system views:
> >> >>
> >> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> >> >> IGNITE-13408 BinaryMetadata view.
> >> >>
> >> >>> 9 окт. 2020 г., в 04:04, Denis Magda < [hidden email] >
> написал(а):
> >> >>>
> >> >>> @Alex Plehanov < [hidden email] >,
> >> >>>
> >> >>> If it still makes sense and not too late, you can cherry-pick the
> >> commit
> >> >>> with the new docs into the 2.9 branch:
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> >> >>>
> >> >>> Anyway, it's not crucial as long as we agreed to make an exception
> for
> >> >> this
> >> >>> release. The docs were already published to the website and
> assembled
> >> >> from
> >> >>> the master. Once the vote passes, I'll make them visible and
> traceable
> >> >> from
> >> >>> the website menus.
> >> >>>
> >> >>> -
> >> >>> Denis
> >> >>>
> >> >>>
> >> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> >> >>  [hidden email] >
> >> >>> wrote:
> >> >>>
> >> >>>> Saikat,
> >> >>>>
> >> >>>> The plan sounds good to me! Do you have an idea for the timeline of
> >> the
> >> >>>> module releases? Let me know if I can help in any way.
> >> >>>>
> >> >>>> Also, I was looking into the modularization IEP and noticed that
> OSGI
> >> >>>> module is discussed to be moved to the extensions, but there is no
> >> >>>> corresponding ticket. Should I create one? I will be happy to help
> >> with
> >> >>>> moving this module to extensions.
> >> >>>>
> >> >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <
> [hidden email]
> >> >:
> >> >>>>
> >> >>>>> Hi Alexey,
> >> >>>>>
> >> >>>>> All the modules as planned in IEP-36
> >> >>>>>
> >> >>>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> >> >>>>> are migrated to ignite-extensions repository.
> >> >>>>>
> >> >>>>> We can definitely plan to release ignite-extensions modules.
> >> >>>>>
> >> >>>>> I have a pending PR related to the kafka module and the PR is in
> >> review
> >> >>>>>  https://github.com/apache/ignite/pull/8222 but its
> corresponding PR
> >> >> has
> >> >>>>> been merged in ignite-extensions repository.
> >> >>>>>
> >> >>>>> My thoughts / action items for the migration efforts and releases
> >> were
> >> >>>>> as follows:
> >> >>>>>
> >> >>>>> 1. Merge the changes for
> https://github.com/apache/ignite/pull/8222
> >> >>>>> 2. Fix the upload nightly packages task for ignite-core to help
> fix
> >> the
> >> >>>>> travis build failure in ignite-extensions repository.
> >> >>>>> 3. Review and update the README.md files for the ignite-extensions
> >> >>>> modules
> >> >>>>> as required.
> >> >>>>> 4. Update the docs for ignite-extensions modules in
> ignite-website.
> >> >>>>> 5. Release each module separately and share updates.
> >> >>>>>
> >> >>>>> Please let me know if you have feedback.
> >> >>>>>
> >> >>>>> Regards,
> >> >>>>> Saikat
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov < [hidden email]
> >
> >> >> wrote:
> >> >>>>>
> >> >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
> >> >>>>>>
> >> >>>>>> I will try to update VCS to GitHub.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <
> [hidden email] >
> >> >>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>> Guys,
> >> >>>>>>>
> >> >>>>>>> I've tried to build release candidate using TC [1]. But:
> >> >>>>>>> 1. I can't choose a branch in this TC task (there is no such
> >> field).
> >> >>>>>>> 2. On the "default" branch I got an error: Failed to collect
> >> changes,
> >> >>>>>>> error: List remote refs failed:
> >> >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> >> >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> >> >>>> connect
> >> >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300,
> parent
> >> >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> >> >>>>>>>
> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master "}
> >> >>>>>>>
> >> >>>>>>> Is there some problem with this TC task? What am I doing wrong?
> >> >>>>>>>
> >> >>>>>>> [1] :
> >> >>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >> >>>>>>>
> >> >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >> >>>>>>  [hidden email] >:
> >> >>>>>>>
> >> >>>>>>>> Saikat, Nikolay,
> >> >>>>>>>>
> >> >>>>>>>> We have migrated a bunch of modules to ignite-extensions
> recently.
> >> >>>>>> Given
> >> >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
> >> (will
> >> >>>>>>>> they?), should we also plan to release the extensions?
> >> >>>>>>>>
> >> >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> >> >>  [hidden email]
> >> >>>>> :
> >> >>>>>>>>
> >> >>>>>>>>> Igniters,
> >> >>>>>>>>>
> >> >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can
> anyone
> >> >>>> review
> >> >>>>>>>> it,
> >> >>>>>>>>> please?
> >> >>>>>>>>>
> >> >>>>>>>>> [1]:  https://github.com/apache/ignite/pull/8273
> >> >>>>>>>>>
> >> >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> >> >>>>  [hidden email]
> >> >>>>>>> :
> >> >>>>>>>>>
> >> >>>>>>>>>> Guys,
> >> >>>>>>>>>>
> >> >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
> >> bug.
> >> >>>>>> This
> >> >>>>>>>>> bug
> >> >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
> >> privately
> >> >>>>>> and
> >> >>>>>>>>>> decided to revert this ticket. I will do it today (after TC
> bot
> >> >>>> visa)
> >> >>>>>>>> if
> >> >>>>>>>>>> there are no objections.
> >> >>>>>>>>>>
> >> >>>>>>>>>> [1]:  https://issues.apache.org/jira/browse/IGNITE-13465
> >> >>>>>>>>>> [2]:  https://issues.apache.org/jira/browse/IGNITE-13134
> >> >>>>>>>>>>
> >> >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> >> >>>>  [hidden email]
> >> >>>>>>> :
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Guys,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> During internal testing, we've found a critical bug with
> >> >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
> >> >>>> sequentially).
> >> >>>>>>>>> This
> >> >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >> >>>>>>>>>>> before release. Under investigation now. I'll let you know
> when
> >> >> we
> >> >>>>>> get
> >> >>>>>>>>>>> something.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <
> [hidden email] >:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> >> version
> >> >>>> of
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> >> generation
> >> >>>> in
> >> >>>>>>>>> later
> >> >>>>>>>>>>>> release, or keep the code clean and fix the regression in
> the
> >> >>>> next
> >> >>>>>>>>> releases?
> >> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> >> fairly
> >> >>>>>>>>>>>> straightforward and does not change the semantics, just
> skips
> >> >> the
> >> >>>>>>>>> factory
> >> >>>>>>>>>>>> closures for certain messages.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
> >> for
> >> >>>> all
> >> >>>>>>>>>>>> workloads (I didn't get any significant drops during
> >> >>>> benchmarking).
> >> >>>>>>>> So
> >> >>>>>>>>>>>> I prefer the runtime generation in later releases.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >> >>>>>>>>>>>> < [hidden email] > wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Alexey, Andrey, Igniters,
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> >> version
> >> >>>> of
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> >> generation
> >> >>>> in
> >> >>>>>>>>> later
> >> >>>>>>>>>>>> release, or keep the code clean and fix the regression in
> the
> >> >>>> next
> >> >>>>>>>>> releases?
> >> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> >> fairly
> >> >>>>>>>>>>>> straightforward and does not change the semantics, just
> skips
> >> >> the
> >> >>>>>>>>> factory
> >> >>>>>>>>>>>> closures for certain messages.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Personally, I would prefer fixing the regression given
> that
> >> we
> >> >>>>>> also
> >> >>>>>>>>>>>> introduced tracing in this release.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >> >>>>>>>>  [hidden email]
> >> >>>>>>>>>> :
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> >> >>>> ed52559eb95
> >> >>>>>>>>> and
> >> >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> >> >>>> average
> >> >>>>>> on
> >> >>>>>>>>> our
> >> >>>>>>>>>>>> environment (about the same results we got benchmarking
> >> 65c30ec6
> >> >>>> vs
> >> >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> >> >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9
> on
> >> >> this
> >> >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> >> >>>> servers, 5
> >> >>>>>>>>> clients
> >> >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> >> >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464
> ns
> >> >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908
> ns
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >> >>>>>>>>>>>>  [hidden email] >:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Hi Everyone,
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> >> >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite
> 2.9,
> >> >> you
> >> >>>>>> can
> >> >>>>>>>>>>>> update the docs by following the instruction.
> Unfortunately, I
> >> >>>>>> won't
> >> >>>>>>>> be
> >> >>>>>>>>>>>> able to spend any time on this project any longer. You can
> >> send
> >> >>>>>> your
> >> >>>>>>>>> pull
> >> >>>>>>>>>>>> requests and questions about the documentation to Denis
> Magda.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> -Artem
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> [1] :
> >> >>>>>>>>>>>>
> >> >>>>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >> >>>>>>>>>>>>  [hidden email] > wrote:
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
> >> >>>>>>>>>>>> unfortunately, I
> >> >>>>>>>>>>>>>>>> cannot spot the difference between old and new
> >> >> implementation
> >> >>>>>> in
> >> >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >> >>>>>>>>>>>> MessageFactoryImpl
> >> >>>>>>>>>>>>>>>> with the old switch statement to the
> >> ignite-2.9-revert-12568
> >> >>>>>>>>> branch
> >> >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >> >>>>>>>> compatible
> >> >>>>>>>>>>>> with the
> >> >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Can you check if this change makes any difference
> >> >>>>>>>> performance-wise
> >> >>>>>>>>>>>> in your
> >> >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
> >> generation
> >> >>>> in
> >> >>>>>>>> the
> >> >>>>>>>>>>>> long
> >> >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
> >> >> factory
> >> >>>>>>>> with
> >> >>>>>>>>>>>> a switch
> >> >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> >> >>>> though,
> >> >>>>>>>>>>>> obviously).
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >> >>>>>>>>>  [hidden email]
> >> >>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Hello guys,
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
> >> [1]),
> >> >>>> it
> >> >>>>>>>>>>>> reduced the
> >> >>>>>>>>>>>>>>>>> drop, but not completely removed it.
> >> >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> >> >>>> patch?
> >> >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> >> >>>> against
> >> >>>>>>>>>>>> 2.8.1
> >> >>>>>>>>>>>>>>>>> release on your environment?
> >> >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3%
> drop
> >> >>>> left,
> >> >>>>>>>>>>>> it's close
> >> >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> >> >>>>>>>>>>>> showstopper. Guys,
> >> >>>>>>>>>>>>>>>>> WDYT?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC
> thin
> >> >>>>>>>> driver
> >> >>>>>>>>>>>> between 2.8
> >> >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker
> and
> >> >>>>>>>> should
> >> >>>>>>>>>>>> be
> >> >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >> >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
> >> (NIO
> >> >>>>>>>>> message
> >> >>>>>>>>>>>>>>>>> send problem in some circumstances). I will
> cherry-pick
> >> it
> >> >>>> if
> >> >>>>>>>>>>>> there is no
> >> >>>>>>>>>>>>>>>>> objection.
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-13411
> >> >>>>>>>>>>>>>>>>> [2]  https://github.com/apache/ignite/pull/8223
> >> >>>>>>>>>>>>>>>>> [3]
> https://issues.apache.org/jira/browse/IGNITE-13414
> >> >>>>>>>>>>>>>>>>> [4]
> https://issues.apache.org/jira/browse/IGNITE-13361
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >> >>>>>>>>  [hidden email]
> >> >>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release.
> Since
> >> >>>>>>>> this
> >> >>>>>>>>>>>> issue is
> >> >>>>>>>>>>>>>>>>>> related to the new master key change functionality
> which
> >> >>>>>>>>>>>> haven't been
> >> >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
> >> commit
> >> >>>>>>>> to
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> release branch.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-13390
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >> >>>>>>>>>>>>  [hidden email] >
> >> >>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Hello, Igniters.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client
> fix
> >> >>>>>>>> [1]
> >> >>>>>>>>>>>> into the
> >> >>>>>>>>>>>>>>>>> 2.9
> >> >>>>>>>>>>>>>>>>>> release
> >> >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> >> >> fields
> >> >>>>>>>>> in
> >> >>>>>>>>>>>> wrong
> >> >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-12809
> >> >>>>>>>>>>>>>>>>>>> [2]
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >> >>>>>>>>>>>>>>>>>  [hidden email] >
> >> >>>>>>>>>>>>>>>>>> написал(а):
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can
> optimize
> >> >>>>>>>>>>>> anything out of
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
> >> ideas
> >> >>>>>>>>>>>> right now),
> >> >>>>>>>>>>>>>>>>> so
> >> >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
> >> the
> >> >>>>>>>>>>>> switch
> >> >>>>>>>>>>>>>>>>> approach
> >> >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
> >> inlining
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> Ignite
> >> >>>>>>>>>>>>>>>>>> messages
> >> >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the
> trick.
> >> Let
> >> >>>>>>>>> me
> >> >>>>>>>>>>>> explore
> >> >>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>> code a bit.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes
> for
> >> >>>>>>>> the
> >> >>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
> >> single
> >> >>>>>>>>>>>> virtual call
> >> >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> >> >>>>>>>> amount
> >> >>>>>>>>>>>> of other
> >> >>>>>>>>>>>>>>>>>> work
> >> >>>>>>>>>>>>>>>>>>>> happening during the message processing.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >> >>>>>>>>>>>>  [hidden email]
> >> >>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >> >>>>>>>>> results.
> >> >>>>>>>>>>>>>>>>> Actually,
> >> >>>>>>>>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >> >>>>>>>>> between
> >> >>>>>>>>>>>> e6a7f93
> >> >>>>>>>>>>>>>>>>>> (first
> >> >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >> >>>>>>>>>>>> 6592dfa5 (last
> >> >>>>>>>>>>>>>>>>>> commit in
> >> >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two
> problematic
> >> >>>>>>>>>>>> commits.
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
> >> for
> >> >>>>>>>> a
> >> >>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>>>> between
> >> >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9
> with
> >> >>>>>>>>>>>> reverted
> >> >>>>>>>>>>>>>>>>>> IGNITE-13060
> >> >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring)
> is
> >> not
> >> >>>>>>>>>>>> related to
> >> >>>>>>>>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some
> performance
> >> >>>>>>>>>>>> problem, and
> >> >>>>>>>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or
> we
> >> >>>>>>>> leave
> >> >>>>>>>>>>>> it as is?
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568
> (MessageFactory
> >> >>>>>>>>>>>> refactoring)?
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >> >>>>>>>>>>>>>>>>>>  [hidden email]
> >> >>>>>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568
> has
> >> an
> >> >>>>>>>>>>>> incorrect fix
> >> >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
> >> branch
> >> >>>>>>>>>>>> [1], so it
> >> >>>>>>>>>>>>>>>>>> cannot be
> >> >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more
> accurate
> >> >>>>>>>> work
> >> >>>>>>>>>>>> with fix
> >> >>>>>>>>>>>>>>>>>> versions
> >> >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >> >>>>>>>>>>>> versions.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> --AG
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> [1]
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >> >>>>>>>>>>>>>>>>>>  [hidden email]
> >> >>>>>>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >> >>>>>>>>>>>>>>>>>  [hidden email]
> >> >>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> Guys,
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060
> and
> >> >>>>>>>>>>>> IGNITE-12568
> >> >>>>>>>>>>>>>>>>>> (reverted
> >> >>>>>>>>>>>>>>>>>>>>>>>> it
> >> >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on
> 2.8.1
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
> >> hot
> >> >>>>>>>>>>>> paths, to
> >> >>>>>>>>>>>>>>>>> trace
> >> >>>>>>>>>>>>>>>>>>>>>>>> these
> >> >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance
> drop
> >> >>>>>>>>> here.
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >> >>>>>>>>> switch/case
> >> >>>>>>>>>>>> block was
> >> >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers.
> The
> >> >>>>>>>>>>>> message factory
> >> >>>>>>>>>>>>>>>>>> is on
> >> >>>>>>>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >> >>>>>>>> impact
> >> >>>>>>>>>>>> on total
> >> >>>>>>>>>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >> >>>>>>>>>>>> microbenchmarks,
> >> >>>>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>>>>>>>> found
> >> >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of
> MessageFactory.create()
> >> >>>>>>>>>>>> about 30-35%
> >> >>>>>>>>>>>>>>>>>> faster
> >> >>>>>>>>>>>>>>>>>>>>>>>> than
> >> >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
> >> switch/case
> >> >>>>>>>>> can
> >> >>>>>>>>>>>>>>>>> effectively
> >> >>>>>>>>>>>>>>>>>>>>>>>> inline
> >> >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >> >>>>>>>> suppliers
> >> >>>>>>>>>>>> relatively
> >> >>>>>>>>>>>>>>>>>> heavy
> >> >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've
> tried to
> >> >>>>>>>>>>>> rewrite the
> >> >>>>>>>>>>>>>>>>> code
> >> >>>>>>>>>>>>>>>>>>>>>>>> using
> >> >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >> >>>>>>>>> interface
> >> >>>>>>>>>>>> (to
> >> >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
> >> "invokevirtual"),
> >> >>>>>>>>>>>> but it gives
> >> >>>>>>>>>>>>>>>>>> back
> >> >>>>>>>>>>>>>>>>>>>>>>>> only
> >> >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case,
> code
> >> >>>>>>>> looks
> >> >>>>>>>>>>>> ugly
> >> >>>>>>>>>>>>>>>>>> (lambdas
> >> >>>>>>>>>>>>>>>>>>>>>>>> can't
> >> >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more
> ways to
> >> >>>>>>>>>>>> optimize the
> >> >>>>>>>>>>>>>>>>>> current
> >> >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case
> block).
> >> >>>>>>>>>>>> Andrey Gura,
> >> >>>>>>>>>>>>>>>>> as
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some
> ideas
> >> >>>>>>>>> about
> >> >>>>>>>>>>>>>>>>>> optimization?
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but
> there
> >> are
> >> >>>>>>>>>>>> some metrics
> >> >>>>>>>>>>>>>>>>>>>>>>>> already
> >> >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
> >> message
> >> >>>>>>>>>>>> factory
> >> >>>>>>>>>>>>>>>>>>>>>>>> implementation
> >> >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements)
> is
> >> >>>>>>>>>>>> already released
> >> >>>>>>>>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message
> factory)
> >> is
> >> >>>>>>>>>>>> only present
> >> >>>>>>>>>>>>>>>>>> in Ignite
> >> >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and
> whichever
> >> new
> >> >>>>>>>>>>>> metrics
> >> >>>>>>>>>>>>>>>>>> created for
> >> >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
> >> unblock
> >> >>>>>>>>>>>> the release
> >> >>>>>>>>>>>>>>>>>> and deal
> >> >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >> >>
> >>
> >>
>

--
Vladislav Pyatkov