[DISCUSS] Ignite 3.0 development approach

[Alexey Goncharuk]

Igniters,

I wanted to pitch a rather radical idea regarding the Ignite 3.0
development which has occurred to me some time ago.

We already have several IEPs targeted to Ignite 3.0 which imply major
changes to the codebase (the change in replication protocol and thus
transactions, change in binary format, updated metastorage, etc). We also
planned significant changes in public APIs: configuration format change,
improvements in cache APIs, SQL APIs, transaction mode rework. The wishlist
of changes for Ignite 3.0 is huge.

So, I was wondering whether it makes sense to try to change the old
codebase, or start with a new baseline and move old pieces of code that do
not require significant rework. Personally, I would go with the second
option for the following reasons:

   - We have a chance to shift the development paradigm in the project and
   introduce the practice of true unit-tests. In the new baseline at the
   beginning there will be no ability to run an end-to-end scenario, thus we
   will be forced to write unit-tests. So far, such practice was hard to
   implement because of tight coupling between Ignite components and inability
   to instantiate components without an instance of KernalContext. For
   example, we should be able to thoroughly test internal primitives, such as
   replication protocol (without actual communication), distributed
   metastorage contracts, persistence layer, etc.
   - We will significantly reduce the development cycle in the beginning
   (right now the RunAll takes two hours of astronomical time with empty TC;
   in the new approach developer will be able to run ALL tests locally in a
   matter of minutes)
   - We can get rid of TC bot and enforce green TC by integrating TC build
   results with GitHub PRs (the same way Travis is currently integrated to PR
   check). We should restrict PR merge without a TC check
   - We will still have to re-write all tests, but only once. If we try to
   modify the old codebase, we would need to modify all the tests for every
   major change (public API change, configuration change)
   - We will have fewer conflicts when working together. For example, I
   cannot imagine how one would merge two changes of getting rid of
   IgniteFuture and changes in replication protocol, for example

Technically, I would suggest creating a new repository for Ignite 3.0
(perhaps, a new clean branch, but a new repo looks nicer to me) and a new
Ignite 3.0 TeamCity project.

While it may seem quite radical, I do believe that this approach will give
us more benefits than trying to make such major changes in the existing
codebase. If needed, let's schedule a discord chat like before to discuss
this.

WDYT?

[Nikolay Izhikov-2]

Hello, Alexey.

I think it will hurt our project more than help.
Developing new features for 2 separate branches with the different APIs and internal structure is overwhelming

Maybe we should relax a bit requirements for Ignite3?
Maybe we should move step by step and make Ignite3 with new configuration than Ignite4 with new transactions, etc?

> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <[hidden email]> написал(а):
>
> Igniters,
>
> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> development which has occurred to me some time ago.
>
> We already have several IEPs targeted to Ignite 3.0 which imply major
> changes to the codebase (the change in replication protocol and thus
> transactions, change in binary format, updated metastorage, etc). We also
> planned significant changes in public APIs: configuration format change,
> improvements in cache APIs, SQL APIs, transaction mode rework. The wishlist
> of changes for Ignite 3.0 is huge.
>
> So, I was wondering whether it makes sense to try to change the old
> codebase, or start with a new baseline and move old pieces of code that do
> not require significant rework. Personally, I would go with the second
> option for the following reasons:
>
>   - We have a chance to shift the development paradigm in the project and
>   introduce the practice of true unit-tests. In the new baseline at the
>   beginning there will be no ability to run an end-to-end scenario, thus we
>   will be forced to write unit-tests. So far, such practice was hard to
>   implement because of tight coupling between Ignite components and inability
>   to instantiate components without an instance of KernalContext. For
>   example, we should be able to thoroughly test internal primitives, such as
>   replication protocol (without actual communication), distributed
>   metastorage contracts, persistence layer, etc.
>   - We will significantly reduce the development cycle in the beginning
>   (right now the RunAll takes two hours of astronomical time with empty TC;
>   in the new approach developer will be able to run ALL tests locally in a
>   matter of minutes)
>   - We can get rid of TC bot and enforce green TC by integrating TC build
>   results with GitHub PRs (the same way Travis is currently integrated to PR
>   check). We should restrict PR merge without a TC check
>   - We will still have to re-write all tests, but only once. If we try to
>   modify the old codebase, we would need to modify all the tests for every
>   major change (public API change, configuration change)
>   - We will have fewer conflicts when working together. For example, I
>   cannot imagine how one would merge two changes of getting rid of
>   IgniteFuture and changes in replication protocol, for example
>
> Technically, I would suggest creating a new repository for Ignite 3.0
> (perhaps, a new clean branch, but a new repo looks nicer to me) and a new
> Ignite 3.0 TeamCity project.
>
> While it may seem quite radical, I do believe that this approach will give
> us more benefits than trying to make such major changes in the existing
> codebase. If needed, let's schedule a discord chat like before to discuss
> this.
>
> WDYT?

[Nikolay Izhikov-2]

To be clear:

> I would suggest creating a new repository for Ignite 3.0 (perhaps, a new clean branch, but a new repo looks nicer to me) and a new Ignite 3.0 TeamCity project.

+1 for new Team City project.
+1 for new branch for Ignite3.
-1 for new repo.

> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]> написал(а):
>
> Hello, Alexey.
>
> I think it will hurt our project more than help.
> Developing new features for 2 separate branches with the different APIs and internal structure is overwhelming
>
> Maybe we should relax a bit requirements for Ignite3?
> Maybe we should move step by step and make Ignite3 with new configuration than Ignite4 with new transactions, etc?
>
>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <[hidden email]> написал(а):
>>
>> Igniters,
>>
>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
>> development which has occurred to me some time ago.
>>
>> We already have several IEPs targeted to Ignite 3.0 which imply major
>> changes to the codebase (the change in replication protocol and thus
>> transactions, change in binary format, updated metastorage, etc). We also
>> planned significant changes in public APIs: configuration format change,
>> improvements in cache APIs, SQL APIs, transaction mode rework. The wishlist
>> of changes for Ignite 3.0 is huge.
>>
>> So, I was wondering whether it makes sense to try to change the old
>> codebase, or start with a new baseline and move old pieces of code that do
>> not require significant rework. Personally, I would go with the second
>> option for the following reasons:
>>
>>  - We have a chance to shift the development paradigm in the project and
>>  introduce the practice of true unit-tests. In the new baseline at the
>>  beginning there will be no ability to run an end-to-end scenario, thus we
>>  will be forced to write unit-tests. So far, such practice was hard to
>>  implement because of tight coupling between Ignite components and inability
>>  to instantiate components without an instance of KernalContext. For
>>  example, we should be able to thoroughly test internal primitives, such as
>>  replication protocol (without actual communication), distributed
>>  metastorage contracts, persistence layer, etc.
>>  - We will significantly reduce the development cycle in the beginning
>>  (right now the RunAll takes two hours of astronomical time with empty TC;
>>  in the new approach developer will be able to run ALL tests locally in a
>>  matter of minutes)
>>  - We can get rid of TC bot and enforce green TC by integrating TC build
>>  results with GitHub PRs (the same way Travis is currently integrated to PR
>>  check). We should restrict PR merge without a TC check
>>  - We will still have to re-write all tests, but only once. If we try to
>>  modify the old codebase, we would need to modify all the tests for every
>>  major change (public API change, configuration change)
>>  - We will have fewer conflicts when working together. For example, I
>>  cannot imagine how one would merge two changes of getting rid of
>>  IgniteFuture and changes in replication protocol, for example
>>
>> Technically, I would suggest creating a new repository for Ignite 3.0
>> (perhaps, a new clean branch, but a new repo looks nicer to me) and a new
>> Ignite 3.0 TeamCity project.
>>
>> While it may seem quite radical, I do believe that this approach will give
>> us more benefits than trying to make such major changes in the existing
>> codebase. If needed, let's schedule a discord chat like before to discuss
>> this.
>>
>> WDYT?
>

[Alexey Goncharuk]

Nikolay,

What new features are we planning to implement for Ignite 2.x? I think once
we commence working on Ignite 3.0, we should gradually cease the activity
on Ignite 2.x to mere bugfixes because such parallel development will be
overwhelming regardless of how we choose to proceed.

пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:

> To be clear:
>
> > I would suggest creating a new repository for Ignite 3.0 (perhaps, a new
> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> TeamCity project.
>
> +1 for new Team City project.
> +1 for new branch for Ignite3.
> -1 for new repo.
>
>
> > 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]>
> написал(а):
> >
> > Hello, Alexey.
> >
> > I think it will hurt our project more than help.
> > Developing new features for 2 separate branches with the different APIs
> and internal structure is overwhelming
> >
> > Maybe we should relax a bit requirements for Ignite3?
> > Maybe we should move step by step and make Ignite3 with new
> configuration than Ignite4 with new transactions, etc?
> >
> >> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <[hidden email]>
> написал(а):
> >>
> >> Igniters,
> >>
> >> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> >> development which has occurred to me some time ago.
> >>
> >> We already have several IEPs targeted to Ignite 3.0 which imply major
> >> changes to the codebase (the change in replication protocol and thus
> >> transactions, change in binary format, updated metastorage, etc). We
> also
> >> planned significant changes in public APIs: configuration format change,
> >> improvements in cache APIs, SQL APIs, transaction mode rework. The
> wishlist
> >> of changes for Ignite 3.0 is huge.
> >>
> >> So, I was wondering whether it makes sense to try to change the old
> >> codebase, or start with a new baseline and move old pieces of code that
> do
> >> not require significant rework. Personally, I would go with the second
> >> option for the following reasons:
> >>
> >>  - We have a chance to shift the development paradigm in the project and
> >>  introduce the practice of true unit-tests. In the new baseline at the
> >>  beginning there will be no ability to run an end-to-end scenario, thus
> we
> >>  will be forced to write unit-tests. So far, such practice was hard to
> >>  implement because of tight coupling between Ignite components and
> inability
> >>  to instantiate components without an instance of KernalContext. For
> >>  example, we should be able to thoroughly test internal primitives,
> such as
> >>  replication protocol (without actual communication), distributed
> >>  metastorage contracts, persistence layer, etc.
> >>  - We will significantly reduce the development cycle in the beginning
> >>  (right now the RunAll takes two hours of astronomical time with empty
> TC;
> >>  in the new approach developer will be able to run ALL tests locally in
> a
> >>  matter of minutes)
> >>  - We can get rid of TC bot and enforce green TC by integrating TC build
> >>  results with GitHub PRs (the same way Travis is currently integrated
> to PR
> >>  check). We should restrict PR merge without a TC check
> >>  - We will still have to re-write all tests, but only once. If we try to
> >>  modify the old codebase, we would need to modify all the tests for
> every
> >>  major change (public API change, configuration change)
> >>  - We will have fewer conflicts when working together. For example, I
> >>  cannot imagine how one would merge two changes of getting rid of
> >>  IgniteFuture and changes in replication protocol, for example
> >>
> >> Technically, I would suggest creating a new repository for Ignite 3.0
> >> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> new
> >> Ignite 3.0 TeamCity project.
> >>
> >> While it may seem quite radical, I do believe that this approach will
> give
> >> us more benefits than trying to make such major changes in the existing
> >> codebase. If needed, let's schedule a discord chat like before to
> discuss
> >> this.
> >>
> >> WDYT?
> >
>
>

[Anton Vinogradov-2]

Alexey,

Do we have any estimates of how fast we'll be able to gain production-ready
AI 3.0 in case of a "new repo" choice?

On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <[hidden email]>
wrote:

> Nikolay,
>
> What new features are we planning to implement for Ignite 2.x? I think once
> we commence working on Ignite 3.0, we should gradually cease the activity
> on Ignite 2.x to mere bugfixes because such parallel development will be
> overwhelming regardless of how we choose to proceed.
>
> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:
>
> > To be clear:
> >
> > > I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> new
> > clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> > TeamCity project.
> >
> > +1 for new Team City project.
> > +1 for new branch for Ignite3.
> > -1 for new repo.
> >
> >
> > > 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]>
> > написал(а):
> > >
> > > Hello, Alexey.
> > >
> > > I think it will hurt our project more than help.
> > > Developing new features for 2 separate branches with the different APIs
> > and internal structure is overwhelming
> > >
> > > Maybe we should relax a bit requirements for Ignite3?
> > > Maybe we should move step by step and make Ignite3 with new
> > configuration than Ignite4 with new transactions, etc?
> > >
> > >> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> [hidden email]>
> > написал(а):
> > >>
> > >> Igniters,
> > >>
> > >> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > >> development which has occurred to me some time ago.
> > >>
> > >> We already have several IEPs targeted to Ignite 3.0 which imply major
> > >> changes to the codebase (the change in replication protocol and thus
> > >> transactions, change in binary format, updated metastorage, etc). We
> > also
> > >> planned significant changes in public APIs: configuration format
> change,
> > >> improvements in cache APIs, SQL APIs, transaction mode rework. The
> > wishlist
> > >> of changes for Ignite 3.0 is huge.
> > >>
> > >> So, I was wondering whether it makes sense to try to change the old
> > >> codebase, or start with a new baseline and move old pieces of code
> that
> > do
> > >> not require significant rework. Personally, I would go with the second
> > >> option for the following reasons:
> > >>
> > >>  - We have a chance to shift the development paradigm in the project
> and
> > >>  introduce the practice of true unit-tests. In the new baseline at the
> > >>  beginning there will be no ability to run an end-to-end scenario,
> thus
> > we
> > >>  will be forced to write unit-tests. So far, such practice was hard to
> > >>  implement because of tight coupling between Ignite components and
> > inability
> > >>  to instantiate components without an instance of KernalContext. For
> > >>  example, we should be able to thoroughly test internal primitives,
> > such as
> > >>  replication protocol (without actual communication), distributed
> > >>  metastorage contracts, persistence layer, etc.
> > >>  - We will significantly reduce the development cycle in the beginning
> > >>  (right now the RunAll takes two hours of astronomical time with empty
> > TC;
> > >>  in the new approach developer will be able to run ALL tests locally
> in
> > a
> > >>  matter of minutes)
> > >>  - We can get rid of TC bot and enforce green TC by integrating TC
> build
> > >>  results with GitHub PRs (the same way Travis is currently integrated
> > to PR
> > >>  check). We should restrict PR merge without a TC check
> > >>  - We will still have to re-write all tests, but only once. If we try
> to
> > >>  modify the old codebase, we would need to modify all the tests for
> > every
> > >>  major change (public API change, configuration change)
> > >>  - We will have fewer conflicts when working together. For example, I
> > >>  cannot imagine how one would merge two changes of getting rid of
> > >>  IgniteFuture and changes in replication protocol, for example
> > >>
> > >> Technically, I would suggest creating a new repository for Ignite 3.0
> > >> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> > new
> > >> Ignite 3.0 TeamCity project.
> > >>
> > >> While it may seem quite radical, I do believe that this approach will
> > give
> > >> us more benefits than trying to make such major changes in the
> existing
> > >> codebase. If needed, let's schedule a discord chat like before to
> > discuss
> > >> this.
> > >>
> > >> WDYT?
> > >
> >
> >
>

[Nikolay Izhikov-2]

Hello, Alexey.

I think that «rewriting from scratch» approach has a high risk to make new features unusable.
At the time Ignite2 was started no-one wants to do bad UX or bad features. Nevertheless, it happen.

I think we can avoid it with the Ignite3 and successors if we will move step by step without keeping backward compatibility
With the step by step approach, we can focus on each component separately.

> What new features are we planning to implement for Ignite 2.x?

We have many features that have to evolve.
Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
We have bugs and issues that can be fixed in 2.x without breaking backward compatibility.
We have many users who are happy with the 2.x with all it’s issues.

> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]> написал(а):
>
> Alexey,
>
> Do we have any estimates of how fast we'll be able to gain production-ready
> AI 3.0 in case of a "new repo" choice?
>
> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <[hidden email]>
> wrote:
>
>> Nikolay,
>>
>> What new features are we planning to implement for Ignite 2.x? I think once
>> we commence working on Ignite 3.0, we should gradually cease the activity
>> on Ignite 2.x to mere bugfixes because such parallel development will be
>> overwhelming regardless of how we choose to proceed.
>>
>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:
>>
>>> To be clear:
>>>
>>>> I would suggest creating a new repository for Ignite 3.0 (perhaps, a
>> new
>>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
>>> TeamCity project.
>>>
>>> +1 for new Team City project.
>>> +1 for new branch for Ignite3.
>>> -1 for new repo.
>>>
>>>
>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]>
>>> написал(а):
>>>>
>>>> Hello, Alexey.
>>>>
>>>> I think it will hurt our project more than help.
>>>> Developing new features for 2 separate branches with the different APIs
>>> and internal structure is overwhelming
>>>>
>>>> Maybe we should relax a bit requirements for Ignite3?
>>>> Maybe we should move step by step and make Ignite3 with new
>>> configuration than Ignite4 with new transactions, etc?
>>>>
>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>> [hidden email]>
>>> написал(а):
>>>>>
>>>>> Igniters,
>>>>>
>>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
>>>>> development which has occurred to me some time ago.
>>>>>
>>>>> We already have several IEPs targeted to Ignite 3.0 which imply major
>>>>> changes to the codebase (the change in replication protocol and thus
>>>>> transactions, change in binary format, updated metastorage, etc). We
>>> also
>>>>> planned significant changes in public APIs: configuration format
>> change,
>>>>> improvements in cache APIs, SQL APIs, transaction mode rework. The
>>> wishlist
>>>>> of changes for Ignite 3.0 is huge.
>>>>>
>>>>> So, I was wondering whether it makes sense to try to change the old
>>>>> codebase, or start with a new baseline and move old pieces of code
>> that
>>> do
>>>>> not require significant rework. Personally, I would go with the second
>>>>> option for the following reasons:
>>>>>
>>>>> - We have a chance to shift the development paradigm in the project
>> and
>>>>> introduce the practice of true unit-tests. In the new baseline at the
>>>>> beginning there will be no ability to run an end-to-end scenario,
>> thus
>>> we
>>>>> will be forced to write unit-tests. So far, such practice was hard to
>>>>> implement because of tight coupling between Ignite components and
>>> inability
>>>>> to instantiate components without an instance of KernalContext. For
>>>>> example, we should be able to thoroughly test internal primitives,
>>> such as
>>>>> replication protocol (without actual communication), distributed
>>>>> metastorage contracts, persistence layer, etc.
>>>>> - We will significantly reduce the development cycle in the beginning
>>>>> (right now the RunAll takes two hours of astronomical time with empty
>>> TC;
>>>>> in the new approach developer will be able to run ALL tests locally
>> in
>>> a
>>>>> matter of minutes)
>>>>> - We can get rid of TC bot and enforce green TC by integrating TC
>> build
>>>>> results with GitHub PRs (the same way Travis is currently integrated
>>> to PR
>>>>> check). We should restrict PR merge without a TC check
>>>>> - We will still have to re-write all tests, but only once. If we try
>> to
>>>>> modify the old codebase, we would need to modify all the tests for
>>> every
>>>>> major change (public API change, configuration change)
>>>>> - We will have fewer conflicts when working together. For example, I
>>>>> cannot imagine how one would merge two changes of getting rid of
>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>
>>>>> Technically, I would suggest creating a new repository for Ignite 3.0
>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
>>> new
>>>>> Ignite 3.0 TeamCity project.
>>>>>
>>>>> While it may seem quite radical, I do believe that this approach will
>>> give
>>>>> us more benefits than trying to make such major changes in the
>> existing
>>>>> codebase. If needed, let's schedule a discord chat like before to
>>> discuss
>>>>> this.
>>>>>
>>>>> WDYT?
>>>>
>>>
>>>
>>

[Pavel Tupitsyn]

1. Rewriting from scratch is never a good idea.
We don't want to follow the path of Netscape and lose all our users
by the time we have a working 3.0 [1]

2. Not sure about new repo - seems like some pain and no gain, what's the
problem with a branch?

3. We should keep existing integration tests when possible.
We have accumulated a lot of edge case knowledge over the years,
it is not a good idea to send all of that down the drain.
Yes, integration tests are slow, but they are the most valuable.
I think we can move more stuff into nightly runs and have a fast and modern
basic suite.


Alexey, you are much more familiar with the Ignite core codebase than most
of us,
can you please explain in more detail which particular feature, in your
opinion,
mandates this "start from scratch" approach?
Is it really not possible at all to follow a less radical way?


[1]
https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/

On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]> wrote:

> Hello, Alexey.
>
> I think that «rewriting from scratch» approach has a high risk to make new
> features unusable.
> At the time Ignite2 was started no-one wants to do bad UX or bad features.
> Nevertheless, it happen.
>
> I think we can avoid it with the Ignite3 and successors if we will move
> step by step without keeping backward compatibility
> With the step by step approach, we can focus on each component separately.
>
> > What new features are we planning to implement for Ignite 2.x?
>
> We have many features that have to evolve.
> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> We have many users who are happy with the 2.x with all it’s issues.
>
> > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]> написал(а):
> >
> > Alexey,
> >
> > Do we have any estimates of how fast we'll be able to gain
> production-ready
> > AI 3.0 in case of a "new repo" choice?
> >
> > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> [hidden email]>
> > wrote:
> >
> >> Nikolay,
> >>
> >> What new features are we planning to implement for Ignite 2.x? I think
> once
> >> we commence working on Ignite 3.0, we should gradually cease the
> activity
> >> on Ignite 2.x to mere bugfixes because such parallel development will be
> >> overwhelming regardless of how we choose to proceed.
> >>
> >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:
> >>
> >>> To be clear:
> >>>
> >>>> I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> >> new
> >>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> >>> TeamCity project.
> >>>
> >>> +1 for new Team City project.
> >>> +1 for new branch for Ignite3.
> >>> -1 for new repo.
> >>>
> >>>
> >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]>
> >>> написал(а):
> >>>>
> >>>> Hello, Alexey.
> >>>>
> >>>> I think it will hurt our project more than help.
> >>>> Developing new features for 2 separate branches with the different
> APIs
> >>> and internal structure is overwhelming
> >>>>
> >>>> Maybe we should relax a bit requirements for Ignite3?
> >>>> Maybe we should move step by step and make Ignite3 with new
> >>> configuration than Ignite4 with new transactions, etc?
> >>>>
> >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >> [hidden email]>
> >>> написал(а):
> >>>>>
> >>>>> Igniters,
> >>>>>
> >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> >>>>> development which has occurred to me some time ago.
> >>>>>
> >>>>> We already have several IEPs targeted to Ignite 3.0 which imply major
> >>>>> changes to the codebase (the change in replication protocol and thus
> >>>>> transactions, change in binary format, updated metastorage, etc). We
> >>> also
> >>>>> planned significant changes in public APIs: configuration format
> >> change,
> >>>>> improvements in cache APIs, SQL APIs, transaction mode rework. The
> >>> wishlist
> >>>>> of changes for Ignite 3.0 is huge.
> >>>>>
> >>>>> So, I was wondering whether it makes sense to try to change the old
> >>>>> codebase, or start with a new baseline and move old pieces of code
> >> that
> >>> do
> >>>>> not require significant rework. Personally, I would go with the
> second
> >>>>> option for the following reasons:
> >>>>>
> >>>>> - We have a chance to shift the development paradigm in the project
> >> and
> >>>>> introduce the practice of true unit-tests. In the new baseline at the
> >>>>> beginning there will be no ability to run an end-to-end scenario,
> >> thus
> >>> we
> >>>>> will be forced to write unit-tests. So far, such practice was hard to
> >>>>> implement because of tight coupling between Ignite components and
> >>> inability
> >>>>> to instantiate components without an instance of KernalContext. For
> >>>>> example, we should be able to thoroughly test internal primitives,
> >>> such as
> >>>>> replication protocol (without actual communication), distributed
> >>>>> metastorage contracts, persistence layer, etc.
> >>>>> - We will significantly reduce the development cycle in the beginning
> >>>>> (right now the RunAll takes two hours of astronomical time with empty
> >>> TC;
> >>>>> in the new approach developer will be able to run ALL tests locally
> >> in
> >>> a
> >>>>> matter of minutes)
> >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> >> build
> >>>>> results with GitHub PRs (the same way Travis is currently integrated
> >>> to PR
> >>>>> check). We should restrict PR merge without a TC check
> >>>>> - We will still have to re-write all tests, but only once. If we try
> >> to
> >>>>> modify the old codebase, we would need to modify all the tests for
> >>> every
> >>>>> major change (public API change, configuration change)
> >>>>> - We will have fewer conflicts when working together. For example, I
> >>>>> cannot imagine how one would merge two changes of getting rid of
> >>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>
> >>>>> Technically, I would suggest creating a new repository for Ignite 3.0
> >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> >>> new
> >>>>> Ignite 3.0 TeamCity project.
> >>>>>
> >>>>> While it may seem quite radical, I do believe that this approach will
> >>> give
> >>>>> us more benefits than trying to make such major changes in the
> >> existing
> >>>>> codebase. If needed, let's schedule a discord chat like before to
> >>> discuss
> >>>>> this.
> >>>>>
> >>>>> WDYT?
> >>>>
> >>>
> >>>
> >>
>
>

[Alexey Goncharuk]

Nikolay, Pavel,

Thanks for the feedback! First of all, I wanted to stress that I do not
intend to rewrite everything from scratch (I never used this phrase). There
are significant parts of code that would be moved with minimal
modifications. Second, I never said that we will get rid of the old tests
codebase. Every single test from Ignite 2.x should be moved to Ignite 3
regardless of how we choose to proceed.

My point is that for some parts of the code a clean bottom-up
implementation will be cheaper in many ways. Let me give you a few concrete
examples:

   - I think no one can object that we need a cleanly separated persistence
   layer for Ignite. There is a very raw draft IEP for this already. On the
   other hand, I think we also can agree that we need a split-brain resistant
   replication protocol for caches. There is also an IEP for this. Neither of
   the changes is a good fit for 2.x because they are likely to introduce
   breaking changes in the persistence layer, configuration and behavior.
   Additionally, these components are now tightly coupled, so there is no way
   these two changes can be implemented in parallel and then merged together
   easily. So what we will end up with is having to implement these changes
   sequentially, fixing all existing tests twice, and essentially throwing
   away half of the work done because the other part of the change is
   re-implemented
   - Similar example goes with getting rid of IgniteInternalFuture and
   replacing it with CompletableFuture, and any other change that touches the
   asynchronous part of the code.

Third, I do not see how this choice affects the UX of Ignite. The end user
experience must be fixed in the IEP regardless of the development process
and the fact that we have gaps in this area in Ignite 2.x just confirms
that.

Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
my point is that we might agree to have a single development master branch
with 'disassembled' end-to-end functionality for some period of time to
speed up development, and re-assemble the core features after having
submodules tested independently.

Nikolay,
  >We have many features that have to evolve.
  >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
This is not very specific. In the end, resources are limited and we will
not be able to drive both tracks simultaneously, especially after a couple
of features having been implemented for Ignite 3.0. If there are indeed
some major changes that we want to do in Ignite 2.x instead of putting
effort into 3.0 - let's discuss them. I am just not aware of any, that's
why I am eager to move forward with Ignite 3.0.

>We have bugs and issues that can be fixed in 2.x without breaking backward
compatibility.
>We have many users who are happy with the 2.x with all it’s issues.
These changes will be covered by end-to-end tests and migrated to Ignite
3.0, so I see no issues here.

Finally, Anton & Nikolay
I do not have an estimate for this simply because the activity is
community-driven and it depends on the number of people willing to
contribute. With the current pace, I would hope to have an RC of Ignite 3.0
to be ready by the end of 2021. My gut feeling is that by moving with
incremental changes, we will not be able to implement even half of the
wishlist by that time.
I doubt that releasing several major releases with breaking changes will
make Ignite users happy either because each upgrade will cost Ignite users
money, so the fewer major versions we release, the better. Thus my wish to
include all breaking changes in one release.

I'll be now quiet for a while, let's see what other community members think.

пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:

> 1. Rewriting from scratch is never a good idea.
> We don't want to follow the path of Netscape and lose all our users
> by the time we have a working 3.0 [1]
>
> 2. Not sure about new repo - seems like some pain and no gain, what's the
> problem with a branch?
>
> 3. We should keep existing integration tests when possible.
> We have accumulated a lot of edge case knowledge over the years,
> it is not a good idea to send all of that down the drain.
> Yes, integration tests are slow, but they are the most valuable.
> I think we can move more stuff into nightly runs and have a fast and modern
> basic suite.
>
>
> Alexey, you are much more familiar with the Ignite core codebase than most
> of us,
> can you please explain in more detail which particular feature, in your
> opinion,
> mandates this "start from scratch" approach?
> Is it really not possible at all to follow a less radical way?
>
>
> [1]
>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>
> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
> wrote:
>
> > Hello, Alexey.
> >
> > I think that «rewriting from scratch» approach has a high risk to make
> new
> > features unusable.
> > At the time Ignite2 was started no-one wants to do bad UX or bad
> features.
> > Nevertheless, it happen.
> >
> > I think we can avoid it with the Ignite3 and successors if we will move
> > step by step without keeping backward compatibility
> > With the step by step approach, we can focus on each component
> separately.
> >
> > > What new features are we planning to implement for Ignite 2.x?
> >
> > We have many features that have to evolve.
> > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > We have bugs and issues that can be fixed in 2.x without breaking
> backward
> > compatibility.
> > We have many users who are happy with the 2.x with all it’s issues.
> >
> > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]> написал(а):
> > >
> > > Alexey,
> > >
> > > Do we have any estimates of how fast we'll be able to gain
> > production-ready
> > > AI 3.0 in case of a "new repo" choice?
> > >
> > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > [hidden email]>
> > > wrote:
> > >
> > >> Nikolay,
> > >>
> > >> What new features are we planning to implement for Ignite 2.x? I think
> > once
> > >> we commence working on Ignite 3.0, we should gradually cease the
> > activity
> > >> on Ignite 2.x to mere bugfixes because such parallel development will
> be
> > >> overwhelming regardless of how we choose to proceed.
> > >>
> > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:
> > >>
> > >>> To be clear:
> > >>>
> > >>>> I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> > >> new
> > >>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> > >>> TeamCity project.
> > >>>
> > >>> +1 for new Team City project.
> > >>> +1 for new branch for Ignite3.
> > >>> -1 for new repo.
> > >>>
> > >>>
> > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]>
> > >>> написал(а):
> > >>>>
> > >>>> Hello, Alexey.
> > >>>>
> > >>>> I think it will hurt our project more than help.
> > >>>> Developing new features for 2 separate branches with the different
> > APIs
> > >>> and internal structure is overwhelming
> > >>>>
> > >>>> Maybe we should relax a bit requirements for Ignite3?
> > >>>> Maybe we should move step by step and make Ignite3 with new
> > >>> configuration than Ignite4 with new transactions, etc?
> > >>>>
> > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > >> [hidden email]>
> > >>> написал(а):
> > >>>>>
> > >>>>> Igniters,
> > >>>>>
> > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > >>>>> development which has occurred to me some time ago.
> > >>>>>
> > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> major
> > >>>>> changes to the codebase (the change in replication protocol and
> thus
> > >>>>> transactions, change in binary format, updated metastorage, etc).
> We
> > >>> also
> > >>>>> planned significant changes in public APIs: configuration format
> > >> change,
> > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework. The
> > >>> wishlist
> > >>>>> of changes for Ignite 3.0 is huge.
> > >>>>>
> > >>>>> So, I was wondering whether it makes sense to try to change the old
> > >>>>> codebase, or start with a new baseline and move old pieces of code
> > >> that
> > >>> do
> > >>>>> not require significant rework. Personally, I would go with the
> > second
> > >>>>> option for the following reasons:
> > >>>>>
> > >>>>> - We have a chance to shift the development paradigm in the project
> > >> and
> > >>>>> introduce the practice of true unit-tests. In the new baseline at
> the
> > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > >> thus
> > >>> we
> > >>>>> will be forced to write unit-tests. So far, such practice was hard
> to
> > >>>>> implement because of tight coupling between Ignite components and
> > >>> inability
> > >>>>> to instantiate components without an instance of KernalContext. For
> > >>>>> example, we should be able to thoroughly test internal primitives,
> > >>> such as
> > >>>>> replication protocol (without actual communication), distributed
> > >>>>> metastorage contracts, persistence layer, etc.
> > >>>>> - We will significantly reduce the development cycle in the
> beginning
> > >>>>> (right now the RunAll takes two hours of astronomical time with
> empty
> > >>> TC;
> > >>>>> in the new approach developer will be able to run ALL tests locally
> > >> in
> > >>> a
> > >>>>> matter of minutes)
> > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> > >> build
> > >>>>> results with GitHub PRs (the same way Travis is currently
> integrated
> > >>> to PR
> > >>>>> check). We should restrict PR merge without a TC check
> > >>>>> - We will still have to re-write all tests, but only once. If we
> try
> > >> to
> > >>>>> modify the old codebase, we would need to modify all the tests for
> > >>> every
> > >>>>> major change (public API change, configuration change)
> > >>>>> - We will have fewer conflicts when working together. For example,
> I
> > >>>>> cannot imagine how one would merge two changes of getting rid of
> > >>>>> IgniteFuture and changes in replication protocol, for example
> > >>>>>
> > >>>>> Technically, I would suggest creating a new repository for Ignite
> 3.0
> > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> and a
> > >>> new
> > >>>>> Ignite 3.0 TeamCity project.
> > >>>>>
> > >>>>> While it may seem quite radical, I do believe that this approach
> will
> > >>> give
> > >>>>> us more benefits than trying to make such major changes in the
> > >> existing
> > >>>>> codebase. If needed, let's schedule a discord chat like before to
> > >>> discuss
> > >>>>> this.
> > >>>>>
> > >>>>> WDYT?
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
>

[Ilya Kasnacheev]

Hello!

In my opinion, what you are actually proposing is writing a new
IMDG/distributed database.

I'm not sure why one would assume that this new product will be
particularly successful with users. We have some very good developers out
there now, but some of the people who actually wrote Ignite 2.x are no
longer with us. Maybe it will be a new and good IMDG, maybe not. Time has
passed, the bar is now higher than it was before. People might as well
consider moving to a different IMDG not called Ignite, than moving to a
different, experimental IMDG named Ignite 3.0.

Maybe there are some bad parts that we have (such as
persistence format/separation, split-brain protocol, etc), but we have 95
out of 100 parts which are very good and battle tested, and I would prefer
them to be kept intact instead of cherry-picking them one by one into the
new grid.

We have very good distributed compute/peer class loading support, people
don't realize how good it is, but they will notice if it is gone/rewritten
from scratch to "just ok" state instead of "very good".

I think that we should just do a deep refactoring/API change in 3.0, and
leave the rest of stuff to 3.x or 4.0. I also happen to think that the code
we already have is very valuable, and it would be a pity to see it replaced
with cleaner, worse offering.

Regards,
--
Ilya Kasnacheev


пн, 2 нояб. 2020 г. в 18:18, Alexey Goncharuk <[hidden email]>:

> Nikolay, Pavel,
>
> Thanks for the feedback! First of all, I wanted to stress that I do not
> intend to rewrite everything from scratch (I never used this phrase). There
> are significant parts of code that would be moved with minimal
> modifications. Second, I never said that we will get rid of the old tests
> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.
>
> My point is that for some parts of the code a clean bottom-up
> implementation will be cheaper in many ways. Let me give you a few concrete
> examples:
>
>    - I think no one can object that we need a cleanly separated persistence
>    layer for Ignite. There is a very raw draft IEP for this already. On the
>    other hand, I think we also can agree that we need a split-brain
> resistant
>    replication protocol for caches. There is also an IEP for this. Neither
> of
>    the changes is a good fit for 2.x because they are likely to introduce
>    breaking changes in the persistence layer, configuration and behavior.
>    Additionally, these components are now tightly coupled, so there is no
> way
>    these two changes can be implemented in parallel and then merged
> together
>    easily. So what we will end up with is having to implement these changes
>    sequentially, fixing all existing tests twice, and essentially throwing
>    away half of the work done because the other part of the change is
>    re-implemented
>    - Similar example goes with getting rid of IgniteInternalFuture and
>    replacing it with CompletableFuture, and any other change that touches
> the
>    asynchronous part of the code.
>
> Third, I do not see how this choice affects the UX of Ignite. The end user
> experience must be fixed in the IEP regardless of the development process
> and the fact that we have gaps in this area in Ignite 2.x just confirms
> that.
>
> Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
> my point is that we might agree to have a single development master branch
> with 'disassembled' end-to-end functionality for some period of time to
> speed up development, and re-assemble the core features after having
> submodules tested independently.
>
> Nikolay,
>   >We have many features that have to evolve.
>   >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> This is not very specific. In the end, resources are limited and we will
> not be able to drive both tracks simultaneously, especially after a couple
> of features having been implemented for Ignite 3.0. If there are indeed
> some major changes that we want to do in Ignite 2.x instead of putting
> effort into 3.0 - let's discuss them. I am just not aware of any, that's
> why I am eager to move forward with Ignite 3.0.
>
> >We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> >We have many users who are happy with the 2.x with all it’s issues.
> These changes will be covered by end-to-end tests and migrated to Ignite
> 3.0, so I see no issues here.
>
> Finally, Anton & Nikolay
> I do not have an estimate for this simply because the activity is
> community-driven and it depends on the number of people willing to
> contribute. With the current pace, I would hope to have an RC of Ignite 3.0
> to be ready by the end of 2021. My gut feeling is that by moving with
> incremental changes, we will not be able to implement even half of the
> wishlist by that time.
> I doubt that releasing several major releases with breaking changes will
> make Ignite users happy either because each upgrade will cost Ignite users
> money, so the fewer major versions we release, the better. Thus my wish to
> include all breaking changes in one release.
>
> I'll be now quiet for a while, let's see what other community members
> think.
>
> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
>
> > 1. Rewriting from scratch is never a good idea.
> > We don't want to follow the path of Netscape and lose all our users
> > by the time we have a working 3.0 [1]
> >
> > 2. Not sure about new repo - seems like some pain and no gain, what's the
> > problem with a branch?
> >
> > 3. We should keep existing integration tests when possible.
> > We have accumulated a lot of edge case knowledge over the years,
> > it is not a good idea to send all of that down the drain.
> > Yes, integration tests are slow, but they are the most valuable.
> > I think we can move more stuff into nightly runs and have a fast and
> modern
> > basic suite.
> >
> >
> > Alexey, you are much more familiar with the Ignite core codebase than
> most
> > of us,
> > can you please explain in more detail which particular feature, in your
> > opinion,
> > mandates this "start from scratch" approach?
> > Is it really not possible at all to follow a less radical way?
> >
> >
> > [1]
> >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >
> > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
> > wrote:
> >
> > > Hello, Alexey.
> > >
> > > I think that «rewriting from scratch» approach has a high risk to make
> > new
> > > features unusable.
> > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > features.
> > > Nevertheless, it happen.
> > >
> > > I think we can avoid it with the Ignite3 and successors if we will move
> > > step by step without keeping backward compatibility
> > > With the step by step approach, we can focus on each component
> > separately.
> > >
> > > > What new features are we planning to implement for Ignite 2.x?
> > >
> > > We have many features that have to evolve.
> > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > We have bugs and issues that can be fixed in 2.x without breaking
> > backward
> > > compatibility.
> > > We have many users who are happy with the 2.x with all it’s issues.
> > >
> > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> написал(а):
> > > >
> > > > Alexey,
> > > >
> > > > Do we have any estimates of how fast we'll be able to gain
> > > production-ready
> > > > AI 3.0 in case of a "new repo" choice?
> > > >
> > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > >> Nikolay,
> > > >>
> > > >> What new features are we planning to implement for Ignite 2.x? I
> think
> > > once
> > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > activity
> > > >> on Ignite 2.x to mere bugfixes because such parallel development
> will
> > be
> > > >> overwhelming regardless of how we choose to proceed.
> > > >>
> > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:
> > > >>
> > > >>> To be clear:
> > > >>>
> > > >>>> I would suggest creating a new repository for Ignite 3.0
> (perhaps, a
> > > >> new
> > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> 3.0
> > > >>> TeamCity project.
> > > >>>
> > > >>> +1 for new Team City project.
> > > >>> +1 for new branch for Ignite3.
> > > >>> -1 for new repo.
> > > >>>
> > > >>>
> > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]
> >
> > > >>> написал(а):
> > > >>>>
> > > >>>> Hello, Alexey.
> > > >>>>
> > > >>>> I think it will hurt our project more than help.
> > > >>>> Developing new features for 2 separate branches with the different
> > > APIs
> > > >>> and internal structure is overwhelming
> > > >>>>
> > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > >>> configuration than Ignite4 with new transactions, etc?
> > > >>>>
> > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >> [hidden email]>
> > > >>> написал(а):
> > > >>>>>
> > > >>>>> Igniters,
> > > >>>>>
> > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > > >>>>> development which has occurred to me some time ago.
> > > >>>>>
> > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> > major
> > > >>>>> changes to the codebase (the change in replication protocol and
> > thus
> > > >>>>> transactions, change in binary format, updated metastorage, etc).
> > We
> > > >>> also
> > > >>>>> planned significant changes in public APIs: configuration format
> > > >> change,
> > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> The
> > > >>> wishlist
> > > >>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>
> > > >>>>> So, I was wondering whether it makes sense to try to change the
> old
> > > >>>>> codebase, or start with a new baseline and move old pieces of
> code
> > > >> that
> > > >>> do
> > > >>>>> not require significant rework. Personally, I would go with the
> > > second
> > > >>>>> option for the following reasons:
> > > >>>>>
> > > >>>>> - We have a chance to shift the development paradigm in the
> project
> > > >> and
> > > >>>>> introduce the practice of true unit-tests. In the new baseline at
> > the
> > > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > > >> thus
> > > >>> we
> > > >>>>> will be forced to write unit-tests. So far, such practice was
> hard
> > to
> > > >>>>> implement because of tight coupling between Ignite components and
> > > >>> inability
> > > >>>>> to instantiate components without an instance of KernalContext.
> For
> > > >>>>> example, we should be able to thoroughly test internal
> primitives,
> > > >>> such as
> > > >>>>> replication protocol (without actual communication), distributed
> > > >>>>> metastorage contracts, persistence layer, etc.
> > > >>>>> - We will significantly reduce the development cycle in the
> > beginning
> > > >>>>> (right now the RunAll takes two hours of astronomical time with
> > empty
> > > >>> TC;
> > > >>>>> in the new approach developer will be able to run ALL tests
> locally
> > > >> in
> > > >>> a
> > > >>>>> matter of minutes)
> > > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> > > >> build
> > > >>>>> results with GitHub PRs (the same way Travis is currently
> > integrated
> > > >>> to PR
> > > >>>>> check). We should restrict PR merge without a TC check
> > > >>>>> - We will still have to re-write all tests, but only once. If we
> > try
> > > >> to
> > > >>>>> modify the old codebase, we would need to modify all the tests
> for
> > > >>> every
> > > >>>>> major change (public API change, configuration change)
> > > >>>>> - We will have fewer conflicts when working together. For
> example,
> > I
> > > >>>>> cannot imagine how one would merge two changes of getting rid of
> > > >>>>> IgniteFuture and changes in replication protocol, for example
> > > >>>>>
> > > >>>>> Technically, I would suggest creating a new repository for Ignite
> > 3.0
> > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> > and a
> > > >>> new
> > > >>>>> Ignite 3.0 TeamCity project.
> > > >>>>>
> > > >>>>> While it may seem quite radical, I do believe that this approach
> > will
> > > >>> give
> > > >>>>> us more benefits than trying to make such major changes in the
> > > >> existing
> > > >>>>> codebase. If needed, let's schedule a discord chat like before to
> > > >>> discuss
> > > >>>>> this.
> > > >>>>>
> > > >>>>> WDYT?
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>

[Andrew Mashenkov]

Hi, Igniters.

* AFAIU, we need a new repo if we want to apply different restrictions to
pull requests,
otherwise I see no difference for myself.
E.g. make static analysis (do we have?), compile, styles, and javadoc
checks mandatory.

I think that relaxed requirements here will lead to bad product quality.

* Agree with Pavel, we should 'keep' integrations tests somehow.
During active development tests will be broken most of time, so,
I'd port them e.g. suite-by-suite once we will have a stable and featured
environment to run them and of course make test's code clear and avoid
bad/non-relevant ones.

* I like bottom-up approach.
With it we could make a better framework. I mean clear component lifecycle,
component wiring mechanics, general methods to approach core components
such as exchange/communication
to avoid code mess like we have in ExchangeFuture with all these custom
callbacks for each component, interfaces like
PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
and so on in various unexpected places.
Hope, we will be able to port most of the good code to the new framework
version.



On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <[hidden email]>
wrote:

> Nikolay, Pavel,
>
> Thanks for the feedback! First of all, I wanted to stress that I do not
> intend to rewrite everything from scratch (I never used this phrase). There
> are significant parts of code that would be moved with minimal
> modifications. Second, I never said that we will get rid of the old tests
> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.
>
> My point is that for some parts of the code a clean bottom-up
> implementation will be cheaper in many ways. Let me give you a few concrete
> examples:
>
>    - I think no one can object that we need a cleanly separated persistence
>    layer for Ignite. There is a very raw draft IEP for this already. On the
>    other hand, I think we also can agree that we need a split-brain
> resistant
>    replication protocol for caches. There is also an IEP for this. Neither
> of
>    the changes is a good fit for 2.x because they are likely to introduce
>    breaking changes in the persistence layer, configuration and behavior.
>    Additionally, these components are now tightly coupled, so there is no
> way
>    these two changes can be implemented in parallel and then merged
> together
>    easily. So what we will end up with is having to implement these changes
>    sequentially, fixing all existing tests twice, and essentially throwing
>    away half of the work done because the other part of the change is
>    re-implemented
>    - Similar example goes with getting rid of IgniteInternalFuture and
>    replacing it with CompletableFuture, and any other change that touches
> the
>    asynchronous part of the code.
>
> Third, I do not see how this choice affects the UX of Ignite. The end user
> experience must be fixed in the IEP regardless of the development process
> and the fact that we have gaps in this area in Ignite 2.x just confirms
> that.
>
> Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
> my point is that we might agree to have a single development master branch
> with 'disassembled' end-to-end functionality for some period of time to
> speed up development, and re-assemble the core features after having
> submodules tested independently.
>
> Nikolay,
>   >We have many features that have to evolve.
>   >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> This is not very specific. In the end, resources are limited and we will
> not be able to drive both tracks simultaneously, especially after a couple
> of features having been implemented for Ignite 3.0. If there are indeed
> some major changes that we want to do in Ignite 2.x instead of putting
> effort into 3.0 - let's discuss them. I am just not aware of any, that's
> why I am eager to move forward with Ignite 3.0.
>
> >We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> >We have many users who are happy with the 2.x with all it’s issues.
> These changes will be covered by end-to-end tests and migrated to Ignite
> 3.0, so I see no issues here.
>
> Finally, Anton & Nikolay
> I do not have an estimate for this simply because the activity is
> community-driven and it depends on the number of people willing to
> contribute. With the current pace, I would hope to have an RC of Ignite 3.0
> to be ready by the end of 2021. My gut feeling is that by moving with
> incremental changes, we will not be able to implement even half of the
> wishlist by that time.
> I doubt that releasing several major releases with breaking changes will
> make Ignite users happy either because each upgrade will cost Ignite users
> money, so the fewer major versions we release, the better. Thus my wish to
> include all breaking changes in one release.
>
> I'll be now quiet for a while, let's see what other community members
> think.
>
> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
>
> > 1. Rewriting from scratch is never a good idea.
> > We don't want to follow the path of Netscape and lose all our users
> > by the time we have a working 3.0 [1]
> >
> > 2. Not sure about new repo - seems like some pain and no gain, what's the
> > problem with a branch?
> >
> > 3. We should keep existing integration tests when possible.
> > We have accumulated a lot of edge case knowledge over the years,
> > it is not a good idea to send all of that down the drain.
> > Yes, integration tests are slow, but they are the most valuable.
> > I think we can move more stuff into nightly runs and have a fast and
> modern
> > basic suite.
> >
> >
> > Alexey, you are much more familiar with the Ignite core codebase than
> most
> > of us,
> > can you please explain in more detail which particular feature, in your
> > opinion,
> > mandates this "start from scratch" approach?
> > Is it really not possible at all to follow a less radical way?
> >
> >
> > [1]
> >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >
> > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
> > wrote:
> >
> > > Hello, Alexey.
> > >
> > > I think that «rewriting from scratch» approach has a high risk to make
> > new
> > > features unusable.
> > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > features.
> > > Nevertheless, it happen.
> > >
> > > I think we can avoid it with the Ignite3 and successors if we will move
> > > step by step without keeping backward compatibility
> > > With the step by step approach, we can focus on each component
> > separately.
> > >
> > > > What new features are we planning to implement for Ignite 2.x?
> > >
> > > We have many features that have to evolve.
> > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > We have bugs and issues that can be fixed in 2.x without breaking
> > backward
> > > compatibility.
> > > We have many users who are happy with the 2.x with all it’s issues.
> > >
> > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> написал(а):
> > > >
> > > > Alexey,
> > > >
> > > > Do we have any estimates of how fast we'll be able to gain
> > > production-ready
> > > > AI 3.0 in case of a "new repo" choice?
> > > >
> > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > >> Nikolay,
> > > >>
> > > >> What new features are we planning to implement for Ignite 2.x? I
> think
> > > once
> > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > activity
> > > >> on Ignite 2.x to mere bugfixes because such parallel development
> will
> > be
> > > >> overwhelming regardless of how we choose to proceed.
> > > >>
> > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]>:
> > > >>
> > > >>> To be clear:
> > > >>>
> > > >>>> I would suggest creating a new repository for Ignite 3.0
> (perhaps, a
> > > >> new
> > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> 3.0
> > > >>> TeamCity project.
> > > >>>
> > > >>> +1 for new Team City project.
> > > >>> +1 for new branch for Ignite3.
> > > >>> -1 for new repo.
> > > >>>
> > > >>>
> > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <[hidden email]
> >
> > > >>> написал(а):
> > > >>>>
> > > >>>> Hello, Alexey.
> > > >>>>
> > > >>>> I think it will hurt our project more than help.
> > > >>>> Developing new features for 2 separate branches with the different
> > > APIs
> > > >>> and internal structure is overwhelming
> > > >>>>
> > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > >>> configuration than Ignite4 with new transactions, etc?
> > > >>>>
> > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >> [hidden email]>
> > > >>> написал(а):
> > > >>>>>
> > > >>>>> Igniters,
> > > >>>>>
> > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > > >>>>> development which has occurred to me some time ago.
> > > >>>>>
> > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> > major
> > > >>>>> changes to the codebase (the change in replication protocol and
> > thus
> > > >>>>> transactions, change in binary format, updated metastorage, etc).
> > We
> > > >>> also
> > > >>>>> planned significant changes in public APIs: configuration format
> > > >> change,
> > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> The
> > > >>> wishlist
> > > >>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>
> > > >>>>> So, I was wondering whether it makes sense to try to change the
> old
> > > >>>>> codebase, or start with a new baseline and move old pieces of
> code
> > > >> that
> > > >>> do
> > > >>>>> not require significant rework. Personally, I would go with the
> > > second
> > > >>>>> option for the following reasons:
> > > >>>>>
> > > >>>>> - We have a chance to shift the development paradigm in the
> project
> > > >> and
> > > >>>>> introduce the practice of true unit-tests. In the new baseline at
> > the
> > > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > > >> thus
> > > >>> we
> > > >>>>> will be forced to write unit-tests. So far, such practice was
> hard
> > to
> > > >>>>> implement because of tight coupling between Ignite components and
> > > >>> inability
> > > >>>>> to instantiate components without an instance of KernalContext.
> For
> > > >>>>> example, we should be able to thoroughly test internal
> primitives,
> > > >>> such as
> > > >>>>> replication protocol (without actual communication), distributed
> > > >>>>> metastorage contracts, persistence layer, etc.
> > > >>>>> - We will significantly reduce the development cycle in the
> > beginning
> > > >>>>> (right now the RunAll takes two hours of astronomical time with
> > empty
> > > >>> TC;
> > > >>>>> in the new approach developer will be able to run ALL tests
> locally
> > > >> in
> > > >>> a
> > > >>>>> matter of minutes)
> > > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> > > >> build
> > > >>>>> results with GitHub PRs (the same way Travis is currently
> > integrated
> > > >>> to PR
> > > >>>>> check). We should restrict PR merge without a TC check
> > > >>>>> - We will still have to re-write all tests, but only once. If we
> > try
> > > >> to
> > > >>>>> modify the old codebase, we would need to modify all the tests
> for
> > > >>> every
> > > >>>>> major change (public API change, configuration change)
> > > >>>>> - We will have fewer conflicts when working together. For
> example,
> > I
> > > >>>>> cannot imagine how one would merge two changes of getting rid of
> > > >>>>> IgniteFuture and changes in replication protocol, for example
> > > >>>>>
> > > >>>>> Technically, I would suggest creating a new repository for Ignite
> > 3.0
> > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> > and a
> > > >>> new
> > > >>>>> Ignite 3.0 TeamCity project.
> > > >>>>>
> > > >>>>> While it may seem quite radical, I do believe that this approach
> > will
> > > >>> give
> > > >>>>> us more benefits than trying to make such major changes in the
> > > >> existing
> > > >>>>> codebase. If needed, let's schedule a discord chat like before to
> > > >>> discuss
> > > >>>>> this.
> > > >>>>>
> > > >>>>> WDYT?
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>

--
Best regards,
Andrey V. Mashenkov

[Pavel Tupitsyn]

> not intend to rewrite everything from scratch

> Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.

Alexey, thank you for the explanation, this addresses all of my concerns.





On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <[hidden email]>
wrote:

> Hi, Igniters.
>
> * AFAIU, we need a new repo if we want to apply different restrictions to
> pull requests,
> otherwise I see no difference for myself.
> E.g. make static analysis (do we have?), compile, styles, and javadoc
> checks mandatory.
>
> I think that relaxed requirements here will lead to bad product quality.
>
> * Agree with Pavel, we should 'keep' integrations tests somehow.
> During active development tests will be broken most of time, so,
> I'd port them e.g. suite-by-suite once we will have a stable and featured
> environment to run them and of course make test's code clear and avoid
> bad/non-relevant ones.
>
> * I like bottom-up approach.
> With it we could make a better framework. I mean clear component lifecycle,
> component wiring mechanics, general methods to approach core components
> such as exchange/communication
> to avoid code mess like we have in ExchangeFuture with all these custom
> callbacks for each component, interfaces like
> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> and so on in various unexpected places.
> Hope, we will be able to port most of the good code to the new framework
> version.
>
>
>
> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> [hidden email]>
> wrote:
>
> > Nikolay, Pavel,
> >
> > Thanks for the feedback! First of all, I wanted to stress that I do not
> > intend to rewrite everything from scratch (I never used this phrase).
> There
> > are significant parts of code that would be moved with minimal
> > modifications. Second, I never said that we will get rid of the old tests
> > codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> > regardless of how we choose to proceed.
> >
> > My point is that for some parts of the code a clean bottom-up
> > implementation will be cheaper in many ways. Let me give you a few
> concrete
> > examples:
> >
> >    - I think no one can object that we need a cleanly separated
> persistence
> >    layer for Ignite. There is a very raw draft IEP for this already. On
> the
> >    other hand, I think we also can agree that we need a split-brain
> > resistant
> >    replication protocol for caches. There is also an IEP for this.
> Neither
> > of
> >    the changes is a good fit for 2.x because they are likely to introduce
> >    breaking changes in the persistence layer, configuration and behavior.
> >    Additionally, these components are now tightly coupled, so there is no
> > way
> >    these two changes can be implemented in parallel and then merged
> > together
> >    easily. So what we will end up with is having to implement these
> changes
> >    sequentially, fixing all existing tests twice, and essentially
> throwing
> >    away half of the work done because the other part of the change is
> >    re-implemented
> >    - Similar example goes with getting rid of IgniteInternalFuture and
> >    replacing it with CompletableFuture, and any other change that touches
> > the
> >    asynchronous part of the code.
> >
> > Third, I do not see how this choice affects the UX of Ignite. The end
> user
> > experience must be fixed in the IEP regardless of the development process
> > and the fact that we have gaps in this area in Ignite 2.x just confirms
> > that.
> >
> > Pavel, agree that a repo/branch is a technicality, I guess if
> reformulate,
> > my point is that we might agree to have a single development master
> branch
> > with 'disassembled' end-to-end functionality for some period of time to
> > speed up development, and re-assemble the core features after having
> > submodules tested independently.
> >
> > Nikolay,
> >   >We have many features that have to evolve.
> >   >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > This is not very specific. In the end, resources are limited and we will
> > not be able to drive both tracks simultaneously, especially after a
> couple
> > of features having been implemented for Ignite 3.0. If there are indeed
> > some major changes that we want to do in Ignite 2.x instead of putting
> > effort into 3.0 - let's discuss them. I am just not aware of any, that's
> > why I am eager to move forward with Ignite 3.0.
> >
> > >We have bugs and issues that can be fixed in 2.x without breaking
> backward
> > compatibility.
> > >We have many users who are happy with the 2.x with all it’s issues.
> > These changes will be covered by end-to-end tests and migrated to Ignite
> > 3.0, so I see no issues here.
> >
> > Finally, Anton & Nikolay
> > I do not have an estimate for this simply because the activity is
> > community-driven and it depends on the number of people willing to
> > contribute. With the current pace, I would hope to have an RC of Ignite
> 3.0
> > to be ready by the end of 2021. My gut feeling is that by moving with
> > incremental changes, we will not be able to implement even half of the
> > wishlist by that time.
> > I doubt that releasing several major releases with breaking changes will
> > make Ignite users happy either because each upgrade will cost Ignite
> users
> > money, so the fewer major versions we release, the better. Thus my wish
> to
> > include all breaking changes in one release.
> >
> > I'll be now quiet for a while, let's see what other community members
> > think.
> >
> > пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
> >
> > > 1. Rewriting from scratch is never a good idea.
> > > We don't want to follow the path of Netscape and lose all our users
> > > by the time we have a working 3.0 [1]
> > >
> > > 2. Not sure about new repo - seems like some pain and no gain, what's
> the
> > > problem with a branch?
> > >
> > > 3. We should keep existing integration tests when possible.
> > > We have accumulated a lot of edge case knowledge over the years,
> > > it is not a good idea to send all of that down the drain.
> > > Yes, integration tests are slow, but they are the most valuable.
> > > I think we can move more stuff into nightly runs and have a fast and
> > modern
> > > basic suite.
> > >
> > >
> > > Alexey, you are much more familiar with the Ignite core codebase than
> > most
> > > of us,
> > > can you please explain in more detail which particular feature, in your
> > > opinion,
> > > mandates this "start from scratch" approach?
> > > Is it really not possible at all to follow a less radical way?
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >
> > > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
> > > wrote:
> > >
> > > > Hello, Alexey.
> > > >
> > > > I think that «rewriting from scratch» approach has a high risk to
> make
> > > new
> > > > features unusable.
> > > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > > features.
> > > > Nevertheless, it happen.
> > > >
> > > > I think we can avoid it with the Ignite3 and successors if we will
> move
> > > > step by step without keeping backward compatibility
> > > > With the step by step approach, we can focus on each component
> > > separately.
> > > >
> > > > > What new features are we planning to implement for Ignite 2.x?
> > > >
> > > > We have many features that have to evolve.
> > > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > > We have bugs and issues that can be fixed in 2.x without breaking
> > > backward
> > > > compatibility.
> > > > We have many users who are happy with the 2.x with all it’s issues.
> > > >
> > > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> > написал(а):
> > > > >
> > > > > Alexey,
> > > > >
> > > > > Do we have any estimates of how fast we'll be able to gain
> > > > production-ready
> > > > > AI 3.0 in case of a "new repo" choice?
> > > > >
> > > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > >> Nikolay,
> > > > >>
> > > > >> What new features are we planning to implement for Ignite 2.x? I
> > think
> > > > once
> > > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > > activity
> > > > >> on Ignite 2.x to mere bugfixes because such parallel development
> > will
> > > be
> > > > >> overwhelming regardless of how we choose to proceed.
> > > > >>
> > > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]
> >:
> > > > >>
> > > > >>> To be clear:
> > > > >>>
> > > > >>>> I would suggest creating a new repository for Ignite 3.0
> > (perhaps, a
> > > > >> new
> > > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> > 3.0
> > > > >>> TeamCity project.
> > > > >>>
> > > > >>> +1 for new Team City project.
> > > > >>> +1 for new branch for Ignite3.
> > > > >>> -1 for new repo.
> > > > >>>
> > > > >>>
> > > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> [hidden email]
> > >
> > > > >>> написал(а):
> > > > >>>>
> > > > >>>> Hello, Alexey.
> > > > >>>>
> > > > >>>> I think it will hurt our project more than help.
> > > > >>>> Developing new features for 2 separate branches with the
> different
> > > > APIs
> > > > >>> and internal structure is overwhelming
> > > > >>>>
> > > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > > >>> configuration than Ignite4 with new transactions, etc?
> > > > >>>>
> > > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > >> [hidden email]>
> > > > >>> написал(а):
> > > > >>>>>
> > > > >>>>> Igniters,
> > > > >>>>>
> > > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite
> 3.0
> > > > >>>>> development which has occurred to me some time ago.
> > > > >>>>>
> > > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> > > major
> > > > >>>>> changes to the codebase (the change in replication protocol and
> > > thus
> > > > >>>>> transactions, change in binary format, updated metastorage,
> etc).
> > > We
> > > > >>> also
> > > > >>>>> planned significant changes in public APIs: configuration
> format
> > > > >> change,
> > > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> > The
> > > > >>> wishlist
> > > > >>>>> of changes for Ignite 3.0 is huge.
> > > > >>>>>
> > > > >>>>> So, I was wondering whether it makes sense to try to change the
> > old
> > > > >>>>> codebase, or start with a new baseline and move old pieces of
> > code
> > > > >> that
> > > > >>> do
> > > > >>>>> not require significant rework. Personally, I would go with the
> > > > second
> > > > >>>>> option for the following reasons:
> > > > >>>>>
> > > > >>>>> - We have a chance to shift the development paradigm in the
> > project
> > > > >> and
> > > > >>>>> introduce the practice of true unit-tests. In the new baseline
> at
> > > the
> > > > >>>>> beginning there will be no ability to run an end-to-end
> scenario,
> > > > >> thus
> > > > >>> we
> > > > >>>>> will be forced to write unit-tests. So far, such practice was
> > hard
> > > to
> > > > >>>>> implement because of tight coupling between Ignite components
> and
> > > > >>> inability
> > > > >>>>> to instantiate components without an instance of KernalContext.
> > For
> > > > >>>>> example, we should be able to thoroughly test internal
> > primitives,
> > > > >>> such as
> > > > >>>>> replication protocol (without actual communication),
> distributed
> > > > >>>>> metastorage contracts, persistence layer, etc.
> > > > >>>>> - We will significantly reduce the development cycle in the
> > > beginning
> > > > >>>>> (right now the RunAll takes two hours of astronomical time with
> > > empty
> > > > >>> TC;
> > > > >>>>> in the new approach developer will be able to run ALL tests
> > locally
> > > > >> in
> > > > >>> a
> > > > >>>>> matter of minutes)
> > > > >>>>> - We can get rid of TC bot and enforce green TC by integrating
> TC
> > > > >> build
> > > > >>>>> results with GitHub PRs (the same way Travis is currently
> > > integrated
> > > > >>> to PR
> > > > >>>>> check). We should restrict PR merge without a TC check
> > > > >>>>> - We will still have to re-write all tests, but only once. If
> we
> > > try
> > > > >> to
> > > > >>>>> modify the old codebase, we would need to modify all the tests
> > for
> > > > >>> every
> > > > >>>>> major change (public API change, configuration change)
> > > > >>>>> - We will have fewer conflicts when working together. For
> > example,
> > > I
> > > > >>>>> cannot imagine how one would merge two changes of getting rid
> of
> > > > >>>>> IgniteFuture and changes in replication protocol, for example
> > > > >>>>>
> > > > >>>>> Technically, I would suggest creating a new repository for
> Ignite
> > > 3.0
> > > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> > > and a
> > > > >>> new
> > > > >>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>
> > > > >>>>> While it may seem quite radical, I do believe that this
> approach
> > > will
> > > > >>> give
> > > > >>>>> us more benefits than trying to make such major changes in the
> > > > >> existing
> > > > >>>>> codebase. If needed, let's schedule a discord chat like before
> to
> > > > >>> discuss
> > > > >>>>> this.
> > > > >>>>>
> > > > >>>>> WDYT?
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

[Nikolay Izhikov-2]

Igniters, should we have a call for this topic?

> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]> написал(а):
>
>> not intend to rewrite everything from scratch
>
>> Every single test from Ignite 2.x should be moved to Ignite 3
>> regardless of how we choose to proceed.
>
> Alexey, thank you for the explanation, this addresses all of my concerns.
>
>
>
>
>
> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <[hidden email]>
> wrote:
>
>> Hi, Igniters.
>>
>> * AFAIU, we need a new repo if we want to apply different restrictions to
>> pull requests,
>> otherwise I see no difference for myself.
>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>> checks mandatory.
>>
>> I think that relaxed requirements here will lead to bad product quality.
>>
>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>> During active development tests will be broken most of time, so,
>> I'd port them e.g. suite-by-suite once we will have a stable and featured
>> environment to run them and of course make test's code clear and avoid
>> bad/non-relevant ones.
>>
>> * I like bottom-up approach.
>> With it we could make a better framework. I mean clear component lifecycle,
>> component wiring mechanics, general methods to approach core components
>> such as exchange/communication
>> to avoid code mess like we have in ExchangeFuture with all these custom
>> callbacks for each component, interfaces like
>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>> and so on in various unexpected places.
>> Hope, we will be able to port most of the good code to the new framework
>> version.
>>
>>
>>
>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>> [hidden email]>
>> wrote:
>>
>>> Nikolay, Pavel,
>>>
>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>> intend to rewrite everything from scratch (I never used this phrase).
>> There
>>> are significant parts of code that would be moved with minimal
>>> modifications. Second, I never said that we will get rid of the old tests
>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>> regardless of how we choose to proceed.
>>>
>>> My point is that for some parts of the code a clean bottom-up
>>> implementation will be cheaper in many ways. Let me give you a few
>> concrete
>>> examples:
>>>
>>>   - I think no one can object that we need a cleanly separated
>> persistence
>>>   layer for Ignite. There is a very raw draft IEP for this already. On
>> the
>>>   other hand, I think we also can agree that we need a split-brain
>>> resistant
>>>   replication protocol for caches. There is also an IEP for this.
>> Neither
>>> of
>>>   the changes is a good fit for 2.x because they are likely to introduce
>>>   breaking changes in the persistence layer, configuration and behavior.
>>>   Additionally, these components are now tightly coupled, so there is no
>>> way
>>>   these two changes can be implemented in parallel and then merged
>>> together
>>>   easily. So what we will end up with is having to implement these
>> changes
>>>   sequentially, fixing all existing tests twice, and essentially
>> throwing
>>>   away half of the work done because the other part of the change is
>>>   re-implemented
>>>   - Similar example goes with getting rid of IgniteInternalFuture and
>>>   replacing it with CompletableFuture, and any other change that touches
>>> the
>>>   asynchronous part of the code.
>>>
>>> Third, I do not see how this choice affects the UX of Ignite. The end
>> user
>>> experience must be fixed in the IEP regardless of the development process
>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>> that.
>>>
>>> Pavel, agree that a repo/branch is a technicality, I guess if
>> reformulate,
>>> my point is that we might agree to have a single development master
>> branch
>>> with 'disassembled' end-to-end functionality for some period of time to
>>> speed up development, and re-assemble the core features after having
>>> submodules tested independently.
>>>
>>> Nikolay,
>>>> We have many features that have to evolve.
>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>> This is not very specific. In the end, resources are limited and we will
>>> not be able to drive both tracks simultaneously, especially after a
>> couple
>>> of features having been implemented for Ignite 3.0. If there are indeed
>>> some major changes that we want to do in Ignite 2.x instead of putting
>>> effort into 3.0 - let's discuss them. I am just not aware of any, that's
>>> why I am eager to move forward with Ignite 3.0.
>>>
>>>> We have bugs and issues that can be fixed in 2.x without breaking
>> backward
>>> compatibility.
>>>> We have many users who are happy with the 2.x with all it’s issues.
>>> These changes will be covered by end-to-end tests and migrated to Ignite
>>> 3.0, so I see no issues here.
>>>
>>> Finally, Anton & Nikolay
>>> I do not have an estimate for this simply because the activity is
>>> community-driven and it depends on the number of people willing to
>>> contribute. With the current pace, I would hope to have an RC of Ignite
>> 3.0
>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>> incremental changes, we will not be able to implement even half of the
>>> wishlist by that time.
>>> I doubt that releasing several major releases with breaking changes will
>>> make Ignite users happy either because each upgrade will cost Ignite
>> users
>>> money, so the fewer major versions we release, the better. Thus my wish
>> to
>>> include all breaking changes in one release.
>>>
>>> I'll be now quiet for a while, let's see what other community members
>>> think.
>>>
>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
>>>
>>>> 1. Rewriting from scratch is never a good idea.
>>>> We don't want to follow the path of Netscape and lose all our users
>>>> by the time we have a working 3.0 [1]
>>>>
>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>> the
>>>> problem with a branch?
>>>>
>>>> 3. We should keep existing integration tests when possible.
>>>> We have accumulated a lot of edge case knowledge over the years,
>>>> it is not a good idea to send all of that down the drain.
>>>> Yes, integration tests are slow, but they are the most valuable.
>>>> I think we can move more stuff into nightly runs and have a fast and
>>> modern
>>>> basic suite.
>>>>
>>>>
>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>> most
>>>> of us,
>>>> can you please explain in more detail which particular feature, in your
>>>> opinion,
>>>> mandates this "start from scratch" approach?
>>>> Is it really not possible at all to follow a less radical way?
>>>>
>>>>
>>>> [1]
>>>>
>>>>
>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>
>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hello, Alexey.
>>>>>
>>>>> I think that «rewriting from scratch» approach has a high risk to
>> make
>>>> new
>>>>> features unusable.
>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>> features.
>>>>> Nevertheless, it happen.
>>>>>
>>>>> I think we can avoid it with the Ignite3 and successors if we will
>> move
>>>>> step by step without keeping backward compatibility
>>>>> With the step by step approach, we can focus on each component
>>>> separately.
>>>>>
>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>>
>>>>> We have many features that have to evolve.
>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>>
>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
>>> написал(а):
>>>>>>
>>>>>> Alexey,
>>>>>>
>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>> production-ready
>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>> [hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Nikolay,
>>>>>>>
>>>>>>> What new features are we planning to implement for Ignite 2.x? I
>>> think
>>>>> once
>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
>>>>> activity
>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>> will
>>>> be
>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>
>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]
>>> :
>>>>>>>
>>>>>>>> To be clear:
>>>>>>>>
>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
>>> (perhaps, a
>>>>>>> new
>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
>>> 3.0
>>>>>>>> TeamCity project.
>>>>>>>>
>>>>>>>> +1 for new Team City project.
>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>> -1 for new repo.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>> [hidden email]
>>>>
>>>>>>>> написал(а):
>>>>>>>>>
>>>>>>>>> Hello, Alexey.
>>>>>>>>>
>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>> Developing new features for 2 separate branches with the
>> different
>>>>> APIs
>>>>>>>> and internal structure is overwhelming
>>>>>>>>>
>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>>
>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>> [hidden email]>
>>>>>>>> написал(а):
>>>>>>>>>>
>>>>>>>>>> Igniters,
>>>>>>>>>>
>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
>> 3.0
>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>>
>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
>>>> major
>>>>>>>>>> changes to the codebase (the change in replication protocol and
>>>> thus
>>>>>>>>>> transactions, change in binary format, updated metastorage,
>> etc).
>>>> We
>>>>>>>> also
>>>>>>>>>> planned significant changes in public APIs: configuration
>> format
>>>>>>> change,
>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
>>> The
>>>>>>>> wishlist
>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>
>>>>>>>>>> So, I was wondering whether it makes sense to try to change the
>>> old
>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
>>> code
>>>>>>> that
>>>>>>>> do
>>>>>>>>>> not require significant rework. Personally, I would go with the
>>>>> second
>>>>>>>>>> option for the following reasons:
>>>>>>>>>>
>>>>>>>>>> - We have a chance to shift the development paradigm in the
>>> project
>>>>>>> and
>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
>> at
>>>> the
>>>>>>>>>> beginning there will be no ability to run an end-to-end
>> scenario,
>>>>>>> thus
>>>>>>>> we
>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
>>> hard
>>>> to
>>>>>>>>>> implement because of tight coupling between Ignite components
>> and
>>>>>>>> inability
>>>>>>>>>> to instantiate components without an instance of KernalContext.
>>> For
>>>>>>>>>> example, we should be able to thoroughly test internal
>>> primitives,
>>>>>>>> such as
>>>>>>>>>> replication protocol (without actual communication),
>> distributed
>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>> - We will significantly reduce the development cycle in the
>>>> beginning
>>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
>>>> empty
>>>>>>>> TC;
>>>>>>>>>> in the new approach developer will be able to run ALL tests
>>> locally
>>>>>>> in
>>>>>>>> a
>>>>>>>>>> matter of minutes)
>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
>> TC
>>>>>>> build
>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>> integrated
>>>>>>>> to PR
>>>>>>>>>> check). We should restrict PR merge without a TC check
>>>>>>>>>> - We will still have to re-write all tests, but only once. If
>> we
>>>> try
>>>>>>> to
>>>>>>>>>> modify the old codebase, we would need to modify all the tests
>>> for
>>>>>>>> every
>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>> - We will have fewer conflicts when working together. For
>>> example,
>>>> I
>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
>> of
>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>>>>>>
>>>>>>>>>> Technically, I would suggest creating a new repository for
>> Ignite
>>>> 3.0
>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
>>>> and a
>>>>>>>> new
>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>
>>>>>>>>>> While it may seem quite radical, I do believe that this
>> approach
>>>> will
>>>>>>>> give
>>>>>>>>>> us more benefits than trying to make such major changes in the
>>>>>>> existing
>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
>> to
>>>>>>>> discuss
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> WDYT?
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>

[nivanov]

+1 on having a separate repo. Make the work cleaner and more effective.

--
Nikita Ivanov



On Mon, Nov 2, 2020 at 11:09 PM Nikolay Izhikov <[hidden email]> wrote:

> Igniters, should we have a call for this topic?
>
> > 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
> написал(а):
> >
> >> not intend to rewrite everything from scratch
> >
> >> Every single test from Ignite 2.x should be moved to Ignite 3
> >> regardless of how we choose to proceed.
> >
> > Alexey, thank you for the explanation, this addresses all of my concerns.
> >
> >
> >
> >
> >
> > On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> [hidden email]>
> > wrote:
> >
> >> Hi, Igniters.
> >>
> >> * AFAIU, we need a new repo if we want to apply different restrictions
> to
> >> pull requests,
> >> otherwise I see no difference for myself.
> >> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >> checks mandatory.
> >>
> >> I think that relaxed requirements here will lead to bad product quality.
> >>
> >> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >> During active development tests will be broken most of time, so,
> >> I'd port them e.g. suite-by-suite once we will have a stable and
> featured
> >> environment to run them and of course make test's code clear and avoid
> >> bad/non-relevant ones.
> >>
> >> * I like bottom-up approach.
> >> With it we could make a better framework. I mean clear component
> lifecycle,
> >> component wiring mechanics, general methods to approach core components
> >> such as exchange/communication
> >> to avoid code mess like we have in ExchangeFuture with all these custom
> >> callbacks for each component, interfaces like
> >> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >> a pack of
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >> and so on in various unexpected places.
> >> Hope, we will be able to port most of the good code to the new framework
> >> version.
> >>
> >>
> >>
> >> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >> [hidden email]>
> >> wrote:
> >>
> >>> Nikolay, Pavel,
> >>>
> >>> Thanks for the feedback! First of all, I wanted to stress that I do not
> >>> intend to rewrite everything from scratch (I never used this phrase).
> >> There
> >>> are significant parts of code that would be moved with minimal
> >>> modifications. Second, I never said that we will get rid of the old
> tests
> >>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> >>> regardless of how we choose to proceed.
> >>>
> >>> My point is that for some parts of the code a clean bottom-up
> >>> implementation will be cheaper in many ways. Let me give you a few
> >> concrete
> >>> examples:
> >>>
> >>>   - I think no one can object that we need a cleanly separated
> >> persistence
> >>>   layer for Ignite. There is a very raw draft IEP for this already. On
> >> the
> >>>   other hand, I think we also can agree that we need a split-brain
> >>> resistant
> >>>   replication protocol for caches. There is also an IEP for this.
> >> Neither
> >>> of
> >>>   the changes is a good fit for 2.x because they are likely to
> introduce
> >>>   breaking changes in the persistence layer, configuration and
> behavior.
> >>>   Additionally, these components are now tightly coupled, so there is
> no
> >>> way
> >>>   these two changes can be implemented in parallel and then merged
> >>> together
> >>>   easily. So what we will end up with is having to implement these
> >> changes
> >>>   sequentially, fixing all existing tests twice, and essentially
> >> throwing
> >>>   away half of the work done because the other part of the change is
> >>>   re-implemented
> >>>   - Similar example goes with getting rid of IgniteInternalFuture and
> >>>   replacing it with CompletableFuture, and any other change that
> touches
> >>> the
> >>>   asynchronous part of the code.
> >>>
> >>> Third, I do not see how this choice affects the UX of Ignite. The end
> >> user
> >>> experience must be fixed in the IEP regardless of the development
> process
> >>> and the fact that we have gaps in this area in Ignite 2.x just confirms
> >>> that.
> >>>
> >>> Pavel, agree that a repo/branch is a technicality, I guess if
> >> reformulate,
> >>> my point is that we might agree to have a single development master
> >> branch
> >>> with 'disassembled' end-to-end functionality for some period of time to
> >>> speed up development, and re-assemble the core features after having
> >>> submodules tested independently.
> >>>
> >>> Nikolay,
> >>>> We have many features that have to evolve.
> >>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>> This is not very specific. In the end, resources are limited and we
> will
> >>> not be able to drive both tracks simultaneously, especially after a
> >> couple
> >>> of features having been implemented for Ignite 3.0. If there are indeed
> >>> some major changes that we want to do in Ignite 2.x instead of putting
> >>> effort into 3.0 - let's discuss them. I am just not aware of any,
> that's
> >>> why I am eager to move forward with Ignite 3.0.
> >>>
> >>>> We have bugs and issues that can be fixed in 2.x without breaking
> >> backward
> >>> compatibility.
> >>>> We have many users who are happy with the 2.x with all it’s issues.
> >>> These changes will be covered by end-to-end tests and migrated to
> Ignite
> >>> 3.0, so I see no issues here.
> >>>
> >>> Finally, Anton & Nikolay
> >>> I do not have an estimate for this simply because the activity is
> >>> community-driven and it depends on the number of people willing to
> >>> contribute. With the current pace, I would hope to have an RC of Ignite
> >> 3.0
> >>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>> incremental changes, we will not be able to implement even half of the
> >>> wishlist by that time.
> >>> I doubt that releasing several major releases with breaking changes
> will
> >>> make Ignite users happy either because each upgrade will cost Ignite
> >> users
> >>> money, so the fewer major versions we release, the better. Thus my wish
> >> to
> >>> include all breaking changes in one release.
> >>>
> >>> I'll be now quiet for a while, let's see what other community members
> >>> think.
> >>>
> >>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
> >>>
> >>>> 1. Rewriting from scratch is never a good idea.
> >>>> We don't want to follow the path of Netscape and lose all our users
> >>>> by the time we have a working 3.0 [1]
> >>>>
> >>>> 2. Not sure about new repo - seems like some pain and no gain, what's
> >> the
> >>>> problem with a branch?
> >>>>
> >>>> 3. We should keep existing integration tests when possible.
> >>>> We have accumulated a lot of edge case knowledge over the years,
> >>>> it is not a good idea to send all of that down the drain.
> >>>> Yes, integration tests are slow, but they are the most valuable.
> >>>> I think we can move more stuff into nightly runs and have a fast and
> >>> modern
> >>>> basic suite.
> >>>>
> >>>>
> >>>> Alexey, you are much more familiar with the Ignite core codebase than
> >>> most
> >>>> of us,
> >>>> can you please explain in more detail which particular feature, in
> your
> >>>> opinion,
> >>>> mandates this "start from scratch" approach?
> >>>> Is it really not possible at all to follow a less radical way?
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>
> >>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hello, Alexey.
> >>>>>
> >>>>> I think that «rewriting from scratch» approach has a high risk to
> >> make
> >>>> new
> >>>>> features unusable.
> >>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>> features.
> >>>>> Nevertheless, it happen.
> >>>>>
> >>>>> I think we can avoid it with the Ignite3 and successors if we will
> >> move
> >>>>> step by step without keeping backward compatibility
> >>>>> With the step by step approach, we can focus on each component
> >>>> separately.
> >>>>>
> >>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>
> >>>>> We have many features that have to evolve.
> >>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>
> >>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> >>> написал(а):
> >>>>>>
> >>>>>> Alexey,
> >>>>>>
> >>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>> production-ready
> >>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>> [hidden email]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Nikolay,
> >>>>>>>
> >>>>>>> What new features are we planning to implement for Ignite 2.x? I
> >>> think
> >>>>> once
> >>>>>>> we commence working on Ignite 3.0, we should gradually cease the
> >>>>> activity
> >>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>> will
> >>>> be
> >>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>
> >>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]
> >>> :
> >>>>>>>
> >>>>>>>> To be clear:
> >>>>>>>>
> >>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> >>> (perhaps, a
> >>>>>>> new
> >>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
> >>> 3.0
> >>>>>>>> TeamCity project.
> >>>>>>>>
> >>>>>>>> +1 for new Team City project.
> >>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>> -1 for new repo.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >> [hidden email]
> >>>>
> >>>>>>>> написал(а):
> >>>>>>>>>
> >>>>>>>>> Hello, Alexey.
> >>>>>>>>>
> >>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>> Developing new features for 2 separate branches with the
> >> different
> >>>>> APIs
> >>>>>>>> and internal structure is overwhelming
> >>>>>>>>>
> >>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> >>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>
> >>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>> [hidden email]>
> >>>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Igniters,
> >>>>>>>>>>
> >>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> >> 3.0
> >>>>>>>>>> development which has occurred to me some time ago.
> >>>>>>>>>>
> >>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> >>>> major
> >>>>>>>>>> changes to the codebase (the change in replication protocol and
> >>>> thus
> >>>>>>>>>> transactions, change in binary format, updated metastorage,
> >> etc).
> >>>> We
> >>>>>>>> also
> >>>>>>>>>> planned significant changes in public APIs: configuration
> >> format
> >>>>>>> change,
> >>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> >>> The
> >>>>>>>> wishlist
> >>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>
> >>>>>>>>>> So, I was wondering whether it makes sense to try to change the
> >>> old
> >>>>>>>>>> codebase, or start with a new baseline and move old pieces of
> >>> code
> >>>>>>> that
> >>>>>>>> do
> >>>>>>>>>> not require significant rework. Personally, I would go with the
> >>>>> second
> >>>>>>>>>> option for the following reasons:
> >>>>>>>>>>
> >>>>>>>>>> - We have a chance to shift the development paradigm in the
> >>> project
> >>>>>>> and
> >>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
> >> at
> >>>> the
> >>>>>>>>>> beginning there will be no ability to run an end-to-end
> >> scenario,
> >>>>>>> thus
> >>>>>>>> we
> >>>>>>>>>> will be forced to write unit-tests. So far, such practice was
> >>> hard
> >>>> to
> >>>>>>>>>> implement because of tight coupling between Ignite components
> >> and
> >>>>>>>> inability
> >>>>>>>>>> to instantiate components without an instance of KernalContext.
> >>> For
> >>>>>>>>>> example, we should be able to thoroughly test internal
> >>> primitives,
> >>>>>>>> such as
> >>>>>>>>>> replication protocol (without actual communication),
> >> distributed
> >>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>> - We will significantly reduce the development cycle in the
> >>>> beginning
> >>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
> >>>> empty
> >>>>>>>> TC;
> >>>>>>>>>> in the new approach developer will be able to run ALL tests
> >>> locally
> >>>>>>> in
> >>>>>>>> a
> >>>>>>>>>> matter of minutes)
> >>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
> >> TC
> >>>>>>> build
> >>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> >>>> integrated
> >>>>>>>> to PR
> >>>>>>>>>> check). We should restrict PR merge without a TC check
> >>>>>>>>>> - We will still have to re-write all tests, but only once. If
> >> we
> >>>> try
> >>>>>>> to
> >>>>>>>>>> modify the old codebase, we would need to modify all the tests
> >>> for
> >>>>>>>> every
> >>>>>>>>>> major change (public API change, configuration change)
> >>>>>>>>>> - We will have fewer conflicts when working together. For
> >>> example,
> >>>> I
> >>>>>>>>>> cannot imagine how one would merge two changes of getting rid
> >> of
> >>>>>>>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>>>>>>
> >>>>>>>>>> Technically, I would suggest creating a new repository for
> >> Ignite
> >>>> 3.0
> >>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> >>>> and a
> >>>>>>>> new
> >>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> While it may seem quite radical, I do believe that this
> >> approach
> >>>> will
> >>>>>>>> give
> >>>>>>>>>> us more benefits than trying to make such major changes in the
> >>>>>>> existing
> >>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
> >> to
> >>>>>>>> discuss
> >>>>>>>>>> this.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>

[Alexey Goncharuk]

Nikolay,

I am up for the call. I will try to explain my reasoning in greater detail
and will be glad to hear the concerns. Will this Friday, Nov 6th, work?

вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <[hidden email]>:

> Igniters, should we have a call for this topic?
>
> > 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
> написал(а):
> >
> >> not intend to rewrite everything from scratch
> >
> >> Every single test from Ignite 2.x should be moved to Ignite 3
> >> regardless of how we choose to proceed.
> >
> > Alexey, thank you for the explanation, this addresses all of my concerns.
> >
> >
> >
> >
> >
> > On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> [hidden email]>
> > wrote:
> >
> >> Hi, Igniters.
> >>
> >> * AFAIU, we need a new repo if we want to apply different restrictions
> to
> >> pull requests,
> >> otherwise I see no difference for myself.
> >> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >> checks mandatory.
> >>
> >> I think that relaxed requirements here will lead to bad product quality.
> >>
> >> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >> During active development tests will be broken most of time, so,
> >> I'd port them e.g. suite-by-suite once we will have a stable and
> featured
> >> environment to run them and of course make test's code clear and avoid
> >> bad/non-relevant ones.
> >>
> >> * I like bottom-up approach.
> >> With it we could make a better framework. I mean clear component
> lifecycle,
> >> component wiring mechanics, general methods to approach core components
> >> such as exchange/communication
> >> to avoid code mess like we have in ExchangeFuture with all these custom
> >> callbacks for each component, interfaces like
> >> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >> a pack of
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >> and so on in various unexpected places.
> >> Hope, we will be able to port most of the good code to the new framework
> >> version.
> >>
> >>
> >>
> >> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >> [hidden email]>
> >> wrote:
> >>
> >>> Nikolay, Pavel,
> >>>
> >>> Thanks for the feedback! First of all, I wanted to stress that I do not
> >>> intend to rewrite everything from scratch (I never used this phrase).
> >> There
> >>> are significant parts of code that would be moved with minimal
> >>> modifications. Second, I never said that we will get rid of the old
> tests
> >>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> >>> regardless of how we choose to proceed.
> >>>
> >>> My point is that for some parts of the code a clean bottom-up
> >>> implementation will be cheaper in many ways. Let me give you a few
> >> concrete
> >>> examples:
> >>>
> >>>   - I think no one can object that we need a cleanly separated
> >> persistence
> >>>   layer for Ignite. There is a very raw draft IEP for this already. On
> >> the
> >>>   other hand, I think we also can agree that we need a split-brain
> >>> resistant
> >>>   replication protocol for caches. There is also an IEP for this.
> >> Neither
> >>> of
> >>>   the changes is a good fit for 2.x because they are likely to
> introduce
> >>>   breaking changes in the persistence layer, configuration and
> behavior.
> >>>   Additionally, these components are now tightly coupled, so there is
> no
> >>> way
> >>>   these two changes can be implemented in parallel and then merged
> >>> together
> >>>   easily. So what we will end up with is having to implement these
> >> changes
> >>>   sequentially, fixing all existing tests twice, and essentially
> >> throwing
> >>>   away half of the work done because the other part of the change is
> >>>   re-implemented
> >>>   - Similar example goes with getting rid of IgniteInternalFuture and
> >>>   replacing it with CompletableFuture, and any other change that
> touches
> >>> the
> >>>   asynchronous part of the code.
> >>>
> >>> Third, I do not see how this choice affects the UX of Ignite. The end
> >> user
> >>> experience must be fixed in the IEP regardless of the development
> process
> >>> and the fact that we have gaps in this area in Ignite 2.x just confirms
> >>> that.
> >>>
> >>> Pavel, agree that a repo/branch is a technicality, I guess if
> >> reformulate,
> >>> my point is that we might agree to have a single development master
> >> branch
> >>> with 'disassembled' end-to-end functionality for some period of time to
> >>> speed up development, and re-assemble the core features after having
> >>> submodules tested independently.
> >>>
> >>> Nikolay,
> >>>> We have many features that have to evolve.
> >>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>> This is not very specific. In the end, resources are limited and we
> will
> >>> not be able to drive both tracks simultaneously, especially after a
> >> couple
> >>> of features having been implemented for Ignite 3.0. If there are indeed
> >>> some major changes that we want to do in Ignite 2.x instead of putting
> >>> effort into 3.0 - let's discuss them. I am just not aware of any,
> that's
> >>> why I am eager to move forward with Ignite 3.0.
> >>>
> >>>> We have bugs and issues that can be fixed in 2.x without breaking
> >> backward
> >>> compatibility.
> >>>> We have many users who are happy with the 2.x with all it’s issues.
> >>> These changes will be covered by end-to-end tests and migrated to
> Ignite
> >>> 3.0, so I see no issues here.
> >>>
> >>> Finally, Anton & Nikolay
> >>> I do not have an estimate for this simply because the activity is
> >>> community-driven and it depends on the number of people willing to
> >>> contribute. With the current pace, I would hope to have an RC of Ignite
> >> 3.0
> >>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>> incremental changes, we will not be able to implement even half of the
> >>> wishlist by that time.
> >>> I doubt that releasing several major releases with breaking changes
> will
> >>> make Ignite users happy either because each upgrade will cost Ignite
> >> users
> >>> money, so the fewer major versions we release, the better. Thus my wish
> >> to
> >>> include all breaking changes in one release.
> >>>
> >>> I'll be now quiet for a while, let's see what other community members
> >>> think.
> >>>
> >>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
> >>>
> >>>> 1. Rewriting from scratch is never a good idea.
> >>>> We don't want to follow the path of Netscape and lose all our users
> >>>> by the time we have a working 3.0 [1]
> >>>>
> >>>> 2. Not sure about new repo - seems like some pain and no gain, what's
> >> the
> >>>> problem with a branch?
> >>>>
> >>>> 3. We should keep existing integration tests when possible.
> >>>> We have accumulated a lot of edge case knowledge over the years,
> >>>> it is not a good idea to send all of that down the drain.
> >>>> Yes, integration tests are slow, but they are the most valuable.
> >>>> I think we can move more stuff into nightly runs and have a fast and
> >>> modern
> >>>> basic suite.
> >>>>
> >>>>
> >>>> Alexey, you are much more familiar with the Ignite core codebase than
> >>> most
> >>>> of us,
> >>>> can you please explain in more detail which particular feature, in
> your
> >>>> opinion,
> >>>> mandates this "start from scratch" approach?
> >>>> Is it really not possible at all to follow a less radical way?
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>
> >>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hello, Alexey.
> >>>>>
> >>>>> I think that «rewriting from scratch» approach has a high risk to
> >> make
> >>>> new
> >>>>> features unusable.
> >>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>> features.
> >>>>> Nevertheless, it happen.
> >>>>>
> >>>>> I think we can avoid it with the Ignite3 and successors if we will
> >> move
> >>>>> step by step without keeping backward compatibility
> >>>>> With the step by step approach, we can focus on each component
> >>>> separately.
> >>>>>
> >>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>
> >>>>> We have many features that have to evolve.
> >>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>
> >>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> >>> написал(а):
> >>>>>>
> >>>>>> Alexey,
> >>>>>>
> >>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>> production-ready
> >>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>> [hidden email]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Nikolay,
> >>>>>>>
> >>>>>>> What new features are we planning to implement for Ignite 2.x? I
> >>> think
> >>>>> once
> >>>>>>> we commence working on Ignite 3.0, we should gradually cease the
> >>>>> activity
> >>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>> will
> >>>> be
> >>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>
> >>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]
> >>> :
> >>>>>>>
> >>>>>>>> To be clear:
> >>>>>>>>
> >>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> >>> (perhaps, a
> >>>>>>> new
> >>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
> >>> 3.0
> >>>>>>>> TeamCity project.
> >>>>>>>>
> >>>>>>>> +1 for new Team City project.
> >>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>> -1 for new repo.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >> [hidden email]
> >>>>
> >>>>>>>> написал(а):
> >>>>>>>>>
> >>>>>>>>> Hello, Alexey.
> >>>>>>>>>
> >>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>> Developing new features for 2 separate branches with the
> >> different
> >>>>> APIs
> >>>>>>>> and internal structure is overwhelming
> >>>>>>>>>
> >>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> >>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>
> >>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>> [hidden email]>
> >>>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Igniters,
> >>>>>>>>>>
> >>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> >> 3.0
> >>>>>>>>>> development which has occurred to me some time ago.
> >>>>>>>>>>
> >>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> >>>> major
> >>>>>>>>>> changes to the codebase (the change in replication protocol and
> >>>> thus
> >>>>>>>>>> transactions, change in binary format, updated metastorage,
> >> etc).
> >>>> We
> >>>>>>>> also
> >>>>>>>>>> planned significant changes in public APIs: configuration
> >> format
> >>>>>>> change,
> >>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> >>> The
> >>>>>>>> wishlist
> >>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>
> >>>>>>>>>> So, I was wondering whether it makes sense to try to change the
> >>> old
> >>>>>>>>>> codebase, or start with a new baseline and move old pieces of
> >>> code
> >>>>>>> that
> >>>>>>>> do
> >>>>>>>>>> not require significant rework. Personally, I would go with the
> >>>>> second
> >>>>>>>>>> option for the following reasons:
> >>>>>>>>>>
> >>>>>>>>>> - We have a chance to shift the development paradigm in the
> >>> project
> >>>>>>> and
> >>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
> >> at
> >>>> the
> >>>>>>>>>> beginning there will be no ability to run an end-to-end
> >> scenario,
> >>>>>>> thus
> >>>>>>>> we
> >>>>>>>>>> will be forced to write unit-tests. So far, such practice was
> >>> hard
> >>>> to
> >>>>>>>>>> implement because of tight coupling between Ignite components
> >> and
> >>>>>>>> inability
> >>>>>>>>>> to instantiate components without an instance of KernalContext.
> >>> For
> >>>>>>>>>> example, we should be able to thoroughly test internal
> >>> primitives,
> >>>>>>>> such as
> >>>>>>>>>> replication protocol (without actual communication),
> >> distributed
> >>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>> - We will significantly reduce the development cycle in the
> >>>> beginning
> >>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
> >>>> empty
> >>>>>>>> TC;
> >>>>>>>>>> in the new approach developer will be able to run ALL tests
> >>> locally
> >>>>>>> in
> >>>>>>>> a
> >>>>>>>>>> matter of minutes)
> >>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
> >> TC
> >>>>>>> build
> >>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> >>>> integrated
> >>>>>>>> to PR
> >>>>>>>>>> check). We should restrict PR merge without a TC check
> >>>>>>>>>> - We will still have to re-write all tests, but only once. If
> >> we
> >>>> try
> >>>>>>> to
> >>>>>>>>>> modify the old codebase, we would need to modify all the tests
> >>> for
> >>>>>>>> every
> >>>>>>>>>> major change (public API change, configuration change)
> >>>>>>>>>> - We will have fewer conflicts when working together. For
> >>> example,
> >>>> I
> >>>>>>>>>> cannot imagine how one would merge two changes of getting rid
> >> of
> >>>>>>>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>>>>>>
> >>>>>>>>>> Technically, I would suggest creating a new repository for
> >> Ignite
> >>>> 3.0
> >>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> >>>> and a
> >>>>>>>> new
> >>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> While it may seem quite radical, I do believe that this
> >> approach
> >>>> will
> >>>>>>>> give
> >>>>>>>>>> us more benefits than trying to make such major changes in the
> >>>>>>> existing
> >>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
> >> to
> >>>>>>>> discuss
> >>>>>>>>>> this.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>

[Nikolay Izhikov-2]

Time works for me.

> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <[hidden email]> написал(а):
>
> Nikolay,
>
> I am up for the call. I will try to explain my reasoning in greater detail
> and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
>
> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <[hidden email]>:
>
>> Igniters, should we have a call for this topic?
>>
>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
>> написал(а):
>>>
>>>> not intend to rewrite everything from scratch
>>>
>>>> Every single test from Ignite 2.x should be moved to Ignite 3
>>>> regardless of how we choose to proceed.
>>>
>>> Alexey, thank you for the explanation, this addresses all of my concerns.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>> [hidden email]>
>>> wrote:
>>>
>>>> Hi, Igniters.
>>>>
>>>> * AFAIU, we need a new repo if we want to apply different restrictions
>> to
>>>> pull requests,
>>>> otherwise I see no difference for myself.
>>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>>>> checks mandatory.
>>>>
>>>> I think that relaxed requirements here will lead to bad product quality.
>>>>
>>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>>>> During active development tests will be broken most of time, so,
>>>> I'd port them e.g. suite-by-suite once we will have a stable and
>> featured
>>>> environment to run them and of course make test's code clear and avoid
>>>> bad/non-relevant ones.
>>>>
>>>> * I like bottom-up approach.
>>>> With it we could make a better framework. I mean clear component
>> lifecycle,
>>>> component wiring mechanics, general methods to approach core components
>>>> such as exchange/communication
>>>> to avoid code mess like we have in ExchangeFuture with all these custom
>>>> callbacks for each component, interfaces like
>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>>>> a pack of
>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>> and so on in various unexpected places.
>>>> Hope, we will be able to port most of the good code to the new framework
>>>> version.
>>>>
>>>>
>>>>
>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>> [hidden email]>
>>>> wrote:
>>>>
>>>>> Nikolay, Pavel,
>>>>>
>>>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>>>> intend to rewrite everything from scratch (I never used this phrase).
>>>> There
>>>>> are significant parts of code that would be moved with minimal
>>>>> modifications. Second, I never said that we will get rid of the old
>> tests
>>>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>>>> regardless of how we choose to proceed.
>>>>>
>>>>> My point is that for some parts of the code a clean bottom-up
>>>>> implementation will be cheaper in many ways. Let me give you a few
>>>> concrete
>>>>> examples:
>>>>>
>>>>>  - I think no one can object that we need a cleanly separated
>>>> persistence
>>>>>  layer for Ignite. There is a very raw draft IEP for this already. On
>>>> the
>>>>>  other hand, I think we also can agree that we need a split-brain
>>>>> resistant
>>>>>  replication protocol for caches. There is also an IEP for this.
>>>> Neither
>>>>> of
>>>>>  the changes is a good fit for 2.x because they are likely to
>> introduce
>>>>>  breaking changes in the persistence layer, configuration and
>> behavior.
>>>>>  Additionally, these components are now tightly coupled, so there is
>> no
>>>>> way
>>>>>  these two changes can be implemented in parallel and then merged
>>>>> together
>>>>>  easily. So what we will end up with is having to implement these
>>>> changes
>>>>>  sequentially, fixing all existing tests twice, and essentially
>>>> throwing
>>>>>  away half of the work done because the other part of the change is
>>>>>  re-implemented
>>>>>  - Similar example goes with getting rid of IgniteInternalFuture and
>>>>>  replacing it with CompletableFuture, and any other change that
>> touches
>>>>> the
>>>>>  asynchronous part of the code.
>>>>>
>>>>> Third, I do not see how this choice affects the UX of Ignite. The end
>>>> user
>>>>> experience must be fixed in the IEP regardless of the development
>> process
>>>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>>>> that.
>>>>>
>>>>> Pavel, agree that a repo/branch is a technicality, I guess if
>>>> reformulate,
>>>>> my point is that we might agree to have a single development master
>>>> branch
>>>>> with 'disassembled' end-to-end functionality for some period of time to
>>>>> speed up development, and re-assemble the core features after having
>>>>> submodules tested independently.
>>>>>
>>>>> Nikolay,
>>>>>> We have many features that have to evolve.
>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> This is not very specific. In the end, resources are limited and we
>> will
>>>>> not be able to drive both tracks simultaneously, especially after a
>>>> couple
>>>>> of features having been implemented for Ignite 3.0. If there are indeed
>>>>> some major changes that we want to do in Ignite 2.x instead of putting
>>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
>> that's
>>>>> why I am eager to move forward with Ignite 3.0.
>>>>>
>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>> These changes will be covered by end-to-end tests and migrated to
>> Ignite
>>>>> 3.0, so I see no issues here.
>>>>>
>>>>> Finally, Anton & Nikolay
>>>>> I do not have an estimate for this simply because the activity is
>>>>> community-driven and it depends on the number of people willing to
>>>>> contribute. With the current pace, I would hope to have an RC of Ignite
>>>> 3.0
>>>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>>>> incremental changes, we will not be able to implement even half of the
>>>>> wishlist by that time.
>>>>> I doubt that releasing several major releases with breaking changes
>> will
>>>>> make Ignite users happy either because each upgrade will cost Ignite
>>>> users
>>>>> money, so the fewer major versions we release, the better. Thus my wish
>>>> to
>>>>> include all breaking changes in one release.
>>>>>
>>>>> I'll be now quiet for a while, let's see what other community members
>>>>> think.
>>>>>
>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
>>>>>
>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>> We don't want to follow the path of Netscape and lose all our users
>>>>>> by the time we have a working 3.0 [1]
>>>>>>
>>>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>>>> the
>>>>>> problem with a branch?
>>>>>>
>>>>>> 3. We should keep existing integration tests when possible.
>>>>>> We have accumulated a lot of edge case knowledge over the years,
>>>>>> it is not a good idea to send all of that down the drain.
>>>>>> Yes, integration tests are slow, but they are the most valuable.
>>>>>> I think we can move more stuff into nightly runs and have a fast and
>>>>> modern
>>>>>> basic suite.
>>>>>>
>>>>>>
>>>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>>>> most
>>>>>> of us,
>>>>>> can you please explain in more detail which particular feature, in
>> your
>>>>>> opinion,
>>>>>> mandates this "start from scratch" approach?
>>>>>> Is it really not possible at all to follow a less radical way?
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>
>>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello, Alexey.
>>>>>>>
>>>>>>> I think that «rewriting from scratch» approach has a high risk to
>>>> make
>>>>>> new
>>>>>>> features unusable.
>>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>>>> features.
>>>>>>> Nevertheless, it happen.
>>>>>>>
>>>>>>> I think we can avoid it with the Ignite3 and successors if we will
>>>> move
>>>>>>> step by step without keeping backward compatibility
>>>>>>> With the step by step approach, we can focus on each component
>>>>>> separately.
>>>>>>>
>>>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>>>>
>>>>>>> We have many features that have to evolve.
>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>>>> backward
>>>>>>> compatibility.
>>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>>>>
>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
>>>>> написал(а):
>>>>>>>>
>>>>>>>> Alexey,
>>>>>>>>
>>>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>>>> production-ready
>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>>
>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>> [hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Nikolay,
>>>>>>>>>
>>>>>>>>> What new features are we planning to implement for Ignite 2.x? I
>>>>> think
>>>>>>> once
>>>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
>>>>>>> activity
>>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>>>> will
>>>>>> be
>>>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>>>
>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <[hidden email]
>>>>> :
>>>>>>>>>
>>>>>>>>>> To be clear:
>>>>>>>>>>
>>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
>>>>> (perhaps, a
>>>>>>>>> new
>>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
>>>>> 3.0
>>>>>>>>>> TeamCity project.
>>>>>>>>>>
>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>> -1 for new repo.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>>>> [hidden email]
>>>>>>
>>>>>>>>>> написал(а):
>>>>>>>>>>>
>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>
>>>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>>>> Developing new features for 2 separate branches with the
>>>> different
>>>>>>> APIs
>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>>
>>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
>>>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>>>>
>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>>>> [hidden email]>
>>>>>>>>>> написал(а):
>>>>>>>>>>>>
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>
>>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
>>>> 3.0
>>>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>>>>
>>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
>>>>>> major
>>>>>>>>>>>> changes to the codebase (the change in replication protocol and
>>>>>> thus
>>>>>>>>>>>> transactions, change in binary format, updated metastorage,
>>>> etc).
>>>>>> We
>>>>>>>>>> also
>>>>>>>>>>>> planned significant changes in public APIs: configuration
>>>> format
>>>>>>>>> change,
>>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
>>>>> The
>>>>>>>>>> wishlist
>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>>
>>>>>>>>>>>> So, I was wondering whether it makes sense to try to change the
>>>>> old
>>>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
>>>>> code
>>>>>>>>> that
>>>>>>>>>> do
>>>>>>>>>>>> not require significant rework. Personally, I would go with the
>>>>>>> second
>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>>
>>>>>>>>>>>> - We have a chance to shift the development paradigm in the
>>>>> project
>>>>>>>>> and
>>>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
>>>> at
>>>>>> the
>>>>>>>>>>>> beginning there will be no ability to run an end-to-end
>>>> scenario,
>>>>>>>>> thus
>>>>>>>>>> we
>>>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
>>>>> hard
>>>>>> to
>>>>>>>>>>>> implement because of tight coupling between Ignite components
>>>> and
>>>>>>>>>> inability
>>>>>>>>>>>> to instantiate components without an instance of KernalContext.
>>>>> For
>>>>>>>>>>>> example, we should be able to thoroughly test internal
>>>>> primitives,
>>>>>>>>>> such as
>>>>>>>>>>>> replication protocol (without actual communication),
>>>> distributed
>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>>>> - We will significantly reduce the development cycle in the
>>>>>> beginning
>>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
>>>>>> empty
>>>>>>>>>> TC;
>>>>>>>>>>>> in the new approach developer will be able to run ALL tests
>>>>> locally
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
>>>> TC
>>>>>>>>> build
>>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>>>> integrated
>>>>>>>>>> to PR
>>>>>>>>>>>> check). We should restrict PR merge without a TC check
>>>>>>>>>>>> - We will still have to re-write all tests, but only once. If
>>>> we
>>>>>> try
>>>>>>>>> to
>>>>>>>>>>>> modify the old codebase, we would need to modify all the tests
>>>>> for
>>>>>>>>>> every
>>>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>>>> - We will have fewer conflicts when working together. For
>>>>> example,
>>>>>> I
>>>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
>>>> of
>>>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>>>>>>>>
>>>>>>>>>>>> Technically, I would suggest creating a new repository for
>>>> Ignite
>>>>>> 3.0
>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
>>>>>> and a
>>>>>>>>>> new
>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>>
>>>>>>>>>>>> While it may seem quite radical, I do believe that this
>>>> approach
>>>>>> will
>>>>>>>>>> give
>>>>>>>>>>>> us more benefits than trying to make such major changes in the
>>>>>>>>> existing
>>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
>>>> to
>>>>>>>>>> discuss
>>>>>>>>>>>> this.
>>>>>>>>>>>>
>>>>>>>>>>>> WDYT?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>
>>

[Kseniya Romanova]

Let's do this community discussion open. Here's the link on zoom call in
Russian for Friday 6 PM:
https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/

вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <[hidden email]>:

> Time works for me.
>
> > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <[hidden email]>
> написал(а):
> >
> > Nikolay,
> >
> > I am up for the call. I will try to explain my reasoning in greater
> detail
> > and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> >
> > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <[hidden email]>:
> >
> >> Igniters, should we have a call for this topic?
> >>
> >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
> >> написал(а):
> >>>
> >>>> not intend to rewrite everything from scratch
> >>>
> >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> >>>> regardless of how we choose to proceed.
> >>>
> >>> Alexey, thank you for the explanation, this addresses all of my
> concerns.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> >> [hidden email]>
> >>> wrote:
> >>>
> >>>> Hi, Igniters.
> >>>>
> >>>> * AFAIU, we need a new repo if we want to apply different restrictions
> >> to
> >>>> pull requests,
> >>>> otherwise I see no difference for myself.
> >>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >>>> checks mandatory.
> >>>>
> >>>> I think that relaxed requirements here will lead to bad product
> quality.
> >>>>
> >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >>>> During active development tests will be broken most of time, so,
> >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> >> featured
> >>>> environment to run them and of course make test's code clear and avoid
> >>>> bad/non-relevant ones.
> >>>>
> >>>> * I like bottom-up approach.
> >>>> With it we could make a better framework. I mean clear component
> >> lifecycle,
> >>>> component wiring mechanics, general methods to approach core
> components
> >>>> such as exchange/communication
> >>>> to avoid code mess like we have in ExchangeFuture with all these
> custom
> >>>> callbacks for each component, interfaces like
> >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >>>> a pack of
> >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >>>> and so on in various unexpected places.
> >>>> Hope, we will be able to port most of the good code to the new
> framework
> >>>> version.
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >>>> [hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Nikolay, Pavel,
> >>>>>
> >>>>> Thanks for the feedback! First of all, I wanted to stress that I do
> not
> >>>>> intend to rewrite everything from scratch (I never used this phrase).
> >>>> There
> >>>>> are significant parts of code that would be moved with minimal
> >>>>> modifications. Second, I never said that we will get rid of the old
> >> tests
> >>>>> codebase. Every single test from Ignite 2.x should be moved to
> Ignite 3
> >>>>> regardless of how we choose to proceed.
> >>>>>
> >>>>> My point is that for some parts of the code a clean bottom-up
> >>>>> implementation will be cheaper in many ways. Let me give you a few
> >>>> concrete
> >>>>> examples:
> >>>>>
> >>>>>  - I think no one can object that we need a cleanly separated
> >>>> persistence
> >>>>>  layer for Ignite. There is a very raw draft IEP for this already. On
> >>>> the
> >>>>>  other hand, I think we also can agree that we need a split-brain
> >>>>> resistant
> >>>>>  replication protocol for caches. There is also an IEP for this.
> >>>> Neither
> >>>>> of
> >>>>>  the changes is a good fit for 2.x because they are likely to
> >> introduce
> >>>>>  breaking changes in the persistence layer, configuration and
> >> behavior.
> >>>>>  Additionally, these components are now tightly coupled, so there is
> >> no
> >>>>> way
> >>>>>  these two changes can be implemented in parallel and then merged
> >>>>> together
> >>>>>  easily. So what we will end up with is having to implement these
> >>>> changes
> >>>>>  sequentially, fixing all existing tests twice, and essentially
> >>>> throwing
> >>>>>  away half of the work done because the other part of the change is
> >>>>>  re-implemented
> >>>>>  - Similar example goes with getting rid of IgniteInternalFuture and
> >>>>>  replacing it with CompletableFuture, and any other change that
> >> touches
> >>>>> the
> >>>>>  asynchronous part of the code.
> >>>>>
> >>>>> Third, I do not see how this choice affects the UX of Ignite. The end
> >>>> user
> >>>>> experience must be fixed in the IEP regardless of the development
> >> process
> >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> confirms
> >>>>> that.
> >>>>>
> >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> >>>> reformulate,
> >>>>> my point is that we might agree to have a single development master
> >>>> branch
> >>>>> with 'disassembled' end-to-end functionality for some period of time
> to
> >>>>> speed up development, and re-assemble the core features after having
> >>>>> submodules tested independently.
> >>>>>
> >>>>> Nikolay,
> >>>>>> We have many features that have to evolve.
> >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> This is not very specific. In the end, resources are limited and we
> >> will
> >>>>> not be able to drive both tracks simultaneously, especially after a
> >>>> couple
> >>>>> of features having been implemented for Ignite 3.0. If there are
> indeed
> >>>>> some major changes that we want to do in Ignite 2.x instead of
> putting
> >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> >> that's
> >>>>> why I am eager to move forward with Ignite 3.0.
> >>>>>
> >>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>> These changes will be covered by end-to-end tests and migrated to
> >> Ignite
> >>>>> 3.0, so I see no issues here.
> >>>>>
> >>>>> Finally, Anton & Nikolay
> >>>>> I do not have an estimate for this simply because the activity is
> >>>>> community-driven and it depends on the number of people willing to
> >>>>> contribute. With the current pace, I would hope to have an RC of
> Ignite
> >>>> 3.0
> >>>>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>>>> incremental changes, we will not be able to implement even half of
> the
> >>>>> wishlist by that time.
> >>>>> I doubt that releasing several major releases with breaking changes
> >> will
> >>>>> make Ignite users happy either because each upgrade will cost Ignite
> >>>> users
> >>>>> money, so the fewer major versions we release, the better. Thus my
> wish
> >>>> to
> >>>>> include all breaking changes in one release.
> >>>>>
> >>>>> I'll be now quiet for a while, let's see what other community members
> >>>>> think.
> >>>>>
> >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]>:
> >>>>>
> >>>>>> 1. Rewriting from scratch is never a good idea.
> >>>>>> We don't want to follow the path of Netscape and lose all our users
> >>>>>> by the time we have a working 3.0 [1]
> >>>>>>
> >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> what's
> >>>> the
> >>>>>> problem with a branch?
> >>>>>>
> >>>>>> 3. We should keep existing integration tests when possible.
> >>>>>> We have accumulated a lot of edge case knowledge over the years,
> >>>>>> it is not a good idea to send all of that down the drain.
> >>>>>> Yes, integration tests are slow, but they are the most valuable.
> >>>>>> I think we can move more stuff into nightly runs and have a fast and
> >>>>> modern
> >>>>>> basic suite.
> >>>>>>
> >>>>>>
> >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> than
> >>>>> most
> >>>>>> of us,
> >>>>>> can you please explain in more detail which particular feature, in
> >> your
> >>>>>> opinion,
> >>>>>> mandates this "start from scratch" approach?
> >>>>>> Is it really not possible at all to follow a less radical way?
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <[hidden email]
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello, Alexey.
> >>>>>>>
> >>>>>>> I think that «rewriting from scratch» approach has a high risk to
> >>>> make
> >>>>>> new
> >>>>>>> features unusable.
> >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>>>> features.
> >>>>>>> Nevertheless, it happen.
> >>>>>>>
> >>>>>>> I think we can avoid it with the Ignite3 and successors if we will
> >>>> move
> >>>>>>> step by step without keeping backward compatibility
> >>>>>>> With the step by step approach, we can focus on each component
> >>>>>> separately.
> >>>>>>>
> >>>>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>>>
> >>>>>>> We have many features that have to evolve.
> >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>>>> backward
> >>>>>>> compatibility.
> >>>>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>>>
> >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> >>>>> написал(а):
> >>>>>>>>
> >>>>>>>> Alexey,
> >>>>>>>>
> >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>>>> production-ready
> >>>>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>>>
> >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>>>> [hidden email]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Nikolay,
> >>>>>>>>>
> >>>>>>>>> What new features are we planning to implement for Ignite 2.x? I
> >>>>> think
> >>>>>>> once
> >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
> >>>>>>> activity
> >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>>>> will
> >>>>>> be
> >>>>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>>>
> >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> [hidden email]
> >>>>> :
> >>>>>>>>>
> >>>>>>>>>> To be clear:
> >>>>>>>>>>
> >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> >>>>> (perhaps, a
> >>>>>>>>> new
> >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
> >>>>> 3.0
> >>>>>>>>>> TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> +1 for new Team City project.
> >>>>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>>>> -1 for new repo.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >>>> [hidden email]
> >>>>>>
> >>>>>>>>>> написал(а):
> >>>>>>>>>>>
> >>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>
> >>>>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>>>> Developing new features for 2 separate branches with the
> >>>> different
> >>>>>>> APIs
> >>>>>>>>>> and internal structure is overwhelming
> >>>>>>>>>>>
> >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>>>
> >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>>>> [hidden email]>
> >>>>>>>>>> написал(а):
> >>>>>>>>>>>>
> >>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> >>>> 3.0
> >>>>>>>>>>>> development which has occurred to me some time ago.
> >>>>>>>>>>>>
> >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which
> imply
> >>>>>> major
> >>>>>>>>>>>> changes to the codebase (the change in replication protocol
> and
> >>>>>> thus
> >>>>>>>>>>>> transactions, change in binary format, updated metastorage,
> >>>> etc).
> >>>>>> We
> >>>>>>>>>> also
> >>>>>>>>>>>> planned significant changes in public APIs: configuration
> >>>> format
> >>>>>>>>> change,
> >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> >>>>> The
> >>>>>>>>>> wishlist
> >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So, I was wondering whether it makes sense to try to change
> the
> >>>>> old
> >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
> >>>>> code
> >>>>>>>>> that
> >>>>>>>>>> do
> >>>>>>>>>>>> not require significant rework. Personally, I would go with
> the
> >>>>>>> second
> >>>>>>>>>>>> option for the following reasons:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - We have a chance to shift the development paradigm in the
> >>>>> project
> >>>>>>>>> and
> >>>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
> >>>> at
> >>>>>> the
> >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> >>>> scenario,
> >>>>>>>>> thus
> >>>>>>>>>> we
> >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
> >>>>> hard
> >>>>>> to
> >>>>>>>>>>>> implement because of tight coupling between Ignite components
> >>>> and
> >>>>>>>>>> inability
> >>>>>>>>>>>> to instantiate components without an instance of
> KernalContext.
> >>>>> For
> >>>>>>>>>>>> example, we should be able to thoroughly test internal
> >>>>> primitives,
> >>>>>>>>>> such as
> >>>>>>>>>>>> replication protocol (without actual communication),
> >>>> distributed
> >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>>>> - We will significantly reduce the development cycle in the
> >>>>>> beginning
> >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time
> with
> >>>>>> empty
> >>>>>>>>>> TC;
> >>>>>>>>>>>> in the new approach developer will be able to run ALL tests
> >>>>> locally
> >>>>>>>>> in
> >>>>>>>>>> a
> >>>>>>>>>>>> matter of minutes)
> >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
> >>>> TC
> >>>>>>>>> build
> >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> >>>>>> integrated
> >>>>>>>>>> to PR
> >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> >>>>>>>>>>>> - We will still have to re-write all tests, but only once. If
> >>>> we
> >>>>>> try
> >>>>>>>>> to
> >>>>>>>>>>>> modify the old codebase, we would need to modify all the tests
> >>>>> for
> >>>>>>>>>> every
> >>>>>>>>>>>> major change (public API change, configuration change)
> >>>>>>>>>>>> - We will have fewer conflicts when working together. For
> >>>>> example,
> >>>>>> I
> >>>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
> >>>> of
> >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>>>>>>>>
> >>>>>>>>>>>> Technically, I would suggest creating a new repository for
> >>>> Ignite
> >>>>>> 3.0
> >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to
> me)
> >>>>>> and a
> >>>>>>>>>> new
> >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> >>>> approach
> >>>>>> will
> >>>>>>>>>> give
> >>>>>>>>>>>> us more benefits than trying to make such major changes in the
> >>>>>>>>> existing
> >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
> >>>> to
> >>>>>>>>>> discuss
> >>>>>>>>>>>> this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> WDYT?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrey V. Mashenkov
> >>>>
> >>
> >>
>
>

[Pavel Tupitsyn]

Kseniya,

Thanks for scheduling this call.
Do you think we can switch to English if non-Russian speaking community
members decide to join?

On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <[hidden email]>
wrote:

> Let's do this community discussion open. Here's the link on zoom call in
> Russian for Friday 6 PM:
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>
> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <[hidden email]>:
>
> > Time works for me.
> >
> > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <[hidden email]
> >
> > написал(а):
> > >
> > > Nikolay,
> > >
> > > I am up for the call. I will try to explain my reasoning in greater
> > detail
> > > and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> > >
> > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <[hidden email]>:
> > >
> > >> Igniters, should we have a call for this topic?
> > >>
> > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
> > >> написал(а):
> > >>>
> > >>>> not intend to rewrite everything from scratch
> > >>>
> > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > >>>> regardless of how we choose to proceed.
> > >>>
> > >>> Alexey, thank you for the explanation, this addresses all of my
> > concerns.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > >> [hidden email]>
> > >>> wrote:
> > >>>
> > >>>> Hi, Igniters.
> > >>>>
> > >>>> * AFAIU, we need a new repo if we want to apply different
> restrictions
> > >> to
> > >>>> pull requests,
> > >>>> otherwise I see no difference for myself.
> > >>>> E.g. make static analysis (do we have?), compile, styles, and
> javadoc
> > >>>> checks mandatory.
> > >>>>
> > >>>> I think that relaxed requirements here will lead to bad product
> > quality.
> > >>>>
> > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > >>>> During active development tests will be broken most of time, so,
> > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > >> featured
> > >>>> environment to run them and of course make test's code clear and
> avoid
> > >>>> bad/non-relevant ones.
> > >>>>
> > >>>> * I like bottom-up approach.
> > >>>> With it we could make a better framework. I mean clear component
> > >> lifecycle,
> > >>>> component wiring mechanics, general methods to approach core
> > components
> > >>>> such as exchange/communication
> > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > custom
> > >>>> callbacks for each component, interfaces like
> > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > >>>> a pack of
> > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > >>>> and so on in various unexpected places.
> > >>>> Hope, we will be able to port most of the good code to the new
> > framework
> > >>>> version.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > >>>> [hidden email]>
> > >>>> wrote:
> > >>>>
> > >>>>> Nikolay, Pavel,
> > >>>>>
> > >>>>> Thanks for the feedback! First of all, I wanted to stress that I do
> > not
> > >>>>> intend to rewrite everything from scratch (I never used this
> phrase).
> > >>>> There
> > >>>>> are significant parts of code that would be moved with minimal
> > >>>>> modifications. Second, I never said that we will get rid of the old
> > >> tests
> > >>>>> codebase. Every single test from Ignite 2.x should be moved to
> > Ignite 3
> > >>>>> regardless of how we choose to proceed.
> > >>>>>
> > >>>>> My point is that for some parts of the code a clean bottom-up
> > >>>>> implementation will be cheaper in many ways. Let me give you a few
> > >>>> concrete
> > >>>>> examples:
> > >>>>>
> > >>>>>  - I think no one can object that we need a cleanly separated
> > >>>> persistence
> > >>>>>  layer for Ignite. There is a very raw draft IEP for this already.
> On
> > >>>> the
> > >>>>>  other hand, I think we also can agree that we need a split-brain
> > >>>>> resistant
> > >>>>>  replication protocol for caches. There is also an IEP for this.
> > >>>> Neither
> > >>>>> of
> > >>>>>  the changes is a good fit for 2.x because they are likely to
> > >> introduce
> > >>>>>  breaking changes in the persistence layer, configuration and
> > >> behavior.
> > >>>>>  Additionally, these components are now tightly coupled, so there
> is
> > >> no
> > >>>>> way
> > >>>>>  these two changes can be implemented in parallel and then merged
> > >>>>> together
> > >>>>>  easily. So what we will end up with is having to implement these
> > >>>> changes
> > >>>>>  sequentially, fixing all existing tests twice, and essentially
> > >>>> throwing
> > >>>>>  away half of the work done because the other part of the change is
> > >>>>>  re-implemented
> > >>>>>  - Similar example goes with getting rid of IgniteInternalFuture
> and
> > >>>>>  replacing it with CompletableFuture, and any other change that
> > >> touches
> > >>>>> the
> > >>>>>  asynchronous part of the code.
> > >>>>>
> > >>>>> Third, I do not see how this choice affects the UX of Ignite. The
> end
> > >>>> user
> > >>>>> experience must be fixed in the IEP regardless of the development
> > >> process
> > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > confirms
> > >>>>> that.
> > >>>>>
> > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > >>>> reformulate,
> > >>>>> my point is that we might agree to have a single development master
> > >>>> branch
> > >>>>> with 'disassembled' end-to-end functionality for some period of
> time
> > to
> > >>>>> speed up development, and re-assemble the core features after
> having
> > >>>>> submodules tested independently.
> > >>>>>
> > >>>>> Nikolay,
> > >>>>>> We have many features that have to evolve.
> > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > >>>>> This is not very specific. In the end, resources are limited and we
> > >> will
> > >>>>> not be able to drive both tracks simultaneously, especially after a
> > >>>> couple
> > >>>>> of features having been implemented for Ignite 3.0. If there are
> > indeed
> > >>>>> some major changes that we want to do in Ignite 2.x instead of
> > putting
> > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> > >> that's
> > >>>>> why I am eager to move forward with Ignite 3.0.
> > >>>>>
> > >>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> > >>>> backward
> > >>>>> compatibility.
> > >>>>>> We have many users who are happy with the 2.x with all it’s
> issues.
> > >>>>> These changes will be covered by end-to-end tests and migrated to
> > >> Ignite
> > >>>>> 3.0, so I see no issues here.
> > >>>>>
> > >>>>> Finally, Anton & Nikolay
> > >>>>> I do not have an estimate for this simply because the activity is
> > >>>>> community-driven and it depends on the number of people willing to
> > >>>>> contribute. With the current pace, I would hope to have an RC of
> > Ignite
> > >>>> 3.0
> > >>>>> to be ready by the end of 2021. My gut feeling is that by moving
> with
> > >>>>> incremental changes, we will not be able to implement even half of
> > the
> > >>>>> wishlist by that time.
> > >>>>> I doubt that releasing several major releases with breaking changes
> > >> will
> > >>>>> make Ignite users happy either because each upgrade will cost
> Ignite
> > >>>> users
> > >>>>> money, so the fewer major versions we release, the better. Thus my
> > wish
> > >>>> to
> > >>>>> include all breaking changes in one release.
> > >>>>>
> > >>>>> I'll be now quiet for a while, let's see what other community
> members
> > >>>>> think.
> > >>>>>
> > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <[hidden email]
> >:
> > >>>>>
> > >>>>>> 1. Rewriting from scratch is never a good idea.
> > >>>>>> We don't want to follow the path of Netscape and lose all our
> users
> > >>>>>> by the time we have a working 3.0 [1]
> > >>>>>>
> > >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> > what's
> > >>>> the
> > >>>>>> problem with a branch?
> > >>>>>>
> > >>>>>> 3. We should keep existing integration tests when possible.
> > >>>>>> We have accumulated a lot of edge case knowledge over the years,
> > >>>>>> it is not a good idea to send all of that down the drain.
> > >>>>>> Yes, integration tests are slow, but they are the most valuable.
> > >>>>>> I think we can move more stuff into nightly runs and have a fast
> and
> > >>>>> modern
> > >>>>>> basic suite.
> > >>>>>>
> > >>>>>>
> > >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> > than
> > >>>>> most
> > >>>>>> of us,
> > >>>>>> can you please explain in more detail which particular feature, in
> > >> your
> > >>>>>> opinion,
> > >>>>>> mandates this "start from scratch" approach?
> > >>>>>> Is it really not possible at all to follow a less radical way?
> > >>>>>>
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >>>>>>
> > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> [hidden email]
> > >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hello, Alexey.
> > >>>>>>>
> > >>>>>>> I think that «rewriting from scratch» approach has a high risk to
> > >>>> make
> > >>>>>> new
> > >>>>>>> features unusable.
> > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> > >>>>>> features.
> > >>>>>>> Nevertheless, it happen.
> > >>>>>>>
> > >>>>>>> I think we can avoid it with the Ignite3 and successors if we
> will
> > >>>> move
> > >>>>>>> step by step without keeping backward compatibility
> > >>>>>>> With the step by step approach, we can focus on each component
> > >>>>>> separately.
> > >>>>>>>
> > >>>>>>>> What new features are we planning to implement for Ignite 2.x?
> > >>>>>>>
> > >>>>>>> We have many features that have to evolve.
> > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > >>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> > >>>>>> backward
> > >>>>>>> compatibility.
> > >>>>>>> We have many users who are happy with the 2.x with all it’s
> issues.
> > >>>>>>>
> > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> > >>>>> написал(а):
> > >>>>>>>>
> > >>>>>>>> Alexey,
> > >>>>>>>>
> > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > >>>>>>> production-ready
> > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > >>>>>>>>
> > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > >>>>>>> [hidden email]>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Nikolay,
> > >>>>>>>>>
> > >>>>>>>>> What new features are we planning to implement for Ignite 2.x?
> I
> > >>>>> think
> > >>>>>>> once
> > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease
> the
> > >>>>>>> activity
> > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> development
> > >>>>> will
> > >>>>>> be
> > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > >>>>>>>>>
> > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > [hidden email]
> > >>>>> :
> > >>>>>>>>>
> > >>>>>>>>>> To be clear:
> > >>>>>>>>>>
> > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> > >>>>> (perhaps, a
> > >>>>>>>>> new
> > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
> Ignite
> > >>>>> 3.0
> > >>>>>>>>>> TeamCity project.
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for new Team City project.
> > >>>>>>>>>> +1 for new branch for Ignite3.
> > >>>>>>>>>> -1 for new repo.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > >>>> [hidden email]
> > >>>>>>
> > >>>>>>>>>> написал(а):
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hello, Alexey.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think it will hurt our project more than help.
> > >>>>>>>>>>> Developing new features for 2 separate branches with the
> > >>>> different
> > >>>>>>> APIs
> > >>>>>>>>>> and internal structure is overwhelming
> > >>>>>>>>>>>
> > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > >>>>>>>>> [hidden email]>
> > >>>>>>>>>> написал(а):
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> > >>>> 3.0
> > >>>>>>>>>>>> development which has occurred to me some time ago.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which
> > imply
> > >>>>>> major
> > >>>>>>>>>>>> changes to the codebase (the change in replication protocol
> > and
> > >>>>>> thus
> > >>>>>>>>>>>> transactions, change in binary format, updated metastorage,
> > >>>> etc).
> > >>>>>> We
> > >>>>>>>>>> also
> > >>>>>>>>>>>> planned significant changes in public APIs: configuration
> > >>>> format
> > >>>>>>>>> change,
> > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> rework.
> > >>>>> The
> > >>>>>>>>>> wishlist
> > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to change
> > the
> > >>>>> old
> > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces
> of
> > >>>>> code
> > >>>>>>>>> that
> > >>>>>>>>>> do
> > >>>>>>>>>>>> not require significant rework. Personally, I would go with
> > the
> > >>>>>>> second
> > >>>>>>>>>>>> option for the following reasons:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - We have a chance to shift the development paradigm in the
> > >>>>> project
> > >>>>>>>>> and
> > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> baseline
> > >>>> at
> > >>>>>> the
> > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> > >>>> scenario,
> > >>>>>>>>> thus
> > >>>>>>>>>> we
> > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice
> was
> > >>>>> hard
> > >>>>>> to
> > >>>>>>>>>>>> implement because of tight coupling between Ignite
> components
> > >>>> and
> > >>>>>>>>>> inability
> > >>>>>>>>>>>> to instantiate components without an instance of
> > KernalContext.
> > >>>>> For
> > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > >>>>> primitives,
> > >>>>>>>>>> such as
> > >>>>>>>>>>>> replication protocol (without actual communication),
> > >>>> distributed
> > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > >>>>>>>>>>>> - We will significantly reduce the development cycle in the
> > >>>>>> beginning
> > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time
> > with
> > >>>>>> empty
> > >>>>>>>>>> TC;
> > >>>>>>>>>>>> in the new approach developer will be able to run ALL tests
> > >>>>> locally
> > >>>>>>>>> in
> > >>>>>>>>>> a
> > >>>>>>>>>>>> matter of minutes)
> > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> integrating
> > >>>> TC
> > >>>>>>>>> build
> > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> > >>>>>> integrated
> > >>>>>>>>>> to PR
> > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > >>>>>>>>>>>> - We will still have to re-write all tests, but only once.
> If
> > >>>> we
> > >>>>>> try
> > >>>>>>>>> to
> > >>>>>>>>>>>> modify the old codebase, we would need to modify all the
> tests
> > >>>>> for
> > >>>>>>>>>> every
> > >>>>>>>>>>>> major change (public API change, configuration change)
> > >>>>>>>>>>>> - We will have fewer conflicts when working together. For
> > >>>>> example,
> > >>>>>> I
> > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting
> rid
> > >>>> of
> > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> example
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Technically, I would suggest creating a new repository for
> > >>>> Ignite
> > >>>>>> 3.0
> > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to
> > me)
> > >>>>>> and a
> > >>>>>>>>>> new
> > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> > >>>> approach
> > >>>>>> will
> > >>>>>>>>>> give
> > >>>>>>>>>>>> us more benefits than trying to make such major changes in
> the
> > >>>>>>>>> existing
> > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like
> before
> > >>>> to
> > >>>>>>>>>> discuss
> > >>>>>>>>>>>> this.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Andrey V. Mashenkov
> > >>>>
> > >>
> > >>
> >
> >
>

[Kseniya Romanova]

Pavel, all the interesting points will be anyway published here in English
(as the principal "if it's not on devlist it doesn't happened" is still
relevant). This is just a quick call for a group of developers. Later we
can do a separate presentation of idea and discussion in English as we did
for the Ignite 3.0 draft of changes.

вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <[hidden email]>:

> Kseniya,
>
> Thanks for scheduling this call.
> Do you think we can switch to English if non-Russian speaking community
> members decide to join?
>
> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <[hidden email]
> >
> wrote:
>
> > Let's do this community discussion open. Here's the link on zoom call in
> > Russian for Friday 6 PM:
> > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> >
> > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <[hidden email]>:
> >
> > > Time works for me.
> > >
> > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> [hidden email]
> > >
> > > написал(а):
> > > >
> > > > Nikolay,
> > > >
> > > > I am up for the call. I will try to explain my reasoning in greater
> > > detail
> > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> work?
> > > >
> > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <[hidden email]>:
> > > >
> > > >> Igniters, should we have a call for this topic?
> > > >>
> > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
> > > >> написал(а):
> > > >>>
> > > >>>> not intend to rewrite everything from scratch
> > > >>>
> > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > > >>>> regardless of how we choose to proceed.
> > > >>>
> > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > concerns.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > >> [hidden email]>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi, Igniters.
> > > >>>>
> > > >>>> * AFAIU, we need a new repo if we want to apply different
> > restrictions
> > > >> to
> > > >>>> pull requests,
> > > >>>> otherwise I see no difference for myself.
> > > >>>> E.g. make static analysis (do we have?), compile, styles, and
> > javadoc
> > > >>>> checks mandatory.
> > > >>>>
> > > >>>> I think that relaxed requirements here will lead to bad product
> > > quality.
> > > >>>>
> > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > > >>>> During active development tests will be broken most of time, so,
> > > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > > >> featured
> > > >>>> environment to run them and of course make test's code clear and
> > avoid
> > > >>>> bad/non-relevant ones.
> > > >>>>
> > > >>>> * I like bottom-up approach.
> > > >>>> With it we could make a better framework. I mean clear component
> > > >> lifecycle,
> > > >>>> component wiring mechanics, general methods to approach core
> > > components
> > > >>>> such as exchange/communication
> > > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > > custom
> > > >>>> callbacks for each component, interfaces like
> > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > >>>> a pack of
> > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > >>>> and so on in various unexpected places.
> > > >>>> Hope, we will be able to port most of the good code to the new
> > > framework
> > > >>>> version.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > >>>> [hidden email]>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Nikolay, Pavel,
> > > >>>>>
> > > >>>>> Thanks for the feedback! First of all, I wanted to stress that I
> do
> > > not
> > > >>>>> intend to rewrite everything from scratch (I never used this
> > phrase).
> > > >>>> There
> > > >>>>> are significant parts of code that would be moved with minimal
> > > >>>>> modifications. Second, I never said that we will get rid of the
> old
> > > >> tests
> > > >>>>> codebase. Every single test from Ignite 2.x should be moved to
> > > Ignite 3
> > > >>>>> regardless of how we choose to proceed.
> > > >>>>>
> > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > >>>>> implementation will be cheaper in many ways. Let me give you a
> few
> > > >>>> concrete
> > > >>>>> examples:
> > > >>>>>
> > > >>>>>  - I think no one can object that we need a cleanly separated
> > > >>>> persistence
> > > >>>>>  layer for Ignite. There is a very raw draft IEP for this
> already.
> > On
> > > >>>> the
> > > >>>>>  other hand, I think we also can agree that we need a split-brain
> > > >>>>> resistant
> > > >>>>>  replication protocol for caches. There is also an IEP for this.
> > > >>>> Neither
> > > >>>>> of
> > > >>>>>  the changes is a good fit for 2.x because they are likely to
> > > >> introduce
> > > >>>>>  breaking changes in the persistence layer, configuration and
> > > >> behavior.
> > > >>>>>  Additionally, these components are now tightly coupled, so there
> > is
> > > >> no
> > > >>>>> way
> > > >>>>>  these two changes can be implemented in parallel and then merged
> > > >>>>> together
> > > >>>>>  easily. So what we will end up with is having to implement these
> > > >>>> changes
> > > >>>>>  sequentially, fixing all existing tests twice, and essentially
> > > >>>> throwing
> > > >>>>>  away half of the work done because the other part of the change
> is
> > > >>>>>  re-implemented
> > > >>>>>  - Similar example goes with getting rid of IgniteInternalFuture
> > and
> > > >>>>>  replacing it with CompletableFuture, and any other change that
> > > >> touches
> > > >>>>> the
> > > >>>>>  asynchronous part of the code.
> > > >>>>>
> > > >>>>> Third, I do not see how this choice affects the UX of Ignite. The
> > end
> > > >>>> user
> > > >>>>> experience must be fixed in the IEP regardless of the development
> > > >> process
> > > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > > confirms
> > > >>>>> that.
> > > >>>>>
> > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > >>>> reformulate,
> > > >>>>> my point is that we might agree to have a single development
> master
> > > >>>> branch
> > > >>>>> with 'disassembled' end-to-end functionality for some period of
> > time
> > > to
> > > >>>>> speed up development, and re-assemble the core features after
> > having
> > > >>>>> submodules tested independently.
> > > >>>>>
> > > >>>>> Nikolay,
> > > >>>>>> We have many features that have to evolve.
> > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > >>>>> This is not very specific. In the end, resources are limited and
> we
> > > >> will
> > > >>>>> not be able to drive both tracks simultaneously, especially
> after a
> > > >>>> couple
> > > >>>>> of features having been implemented for Ignite 3.0. If there are
> > > indeed
> > > >>>>> some major changes that we want to do in Ignite 2.x instead of
> > > putting
> > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> > > >> that's
> > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > >>>>>
> > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> breaking
> > > >>>> backward
> > > >>>>> compatibility.
> > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > issues.
> > > >>>>> These changes will be covered by end-to-end tests and migrated to
> > > >> Ignite
> > > >>>>> 3.0, so I see no issues here.
> > > >>>>>
> > > >>>>> Finally, Anton & Nikolay
> > > >>>>> I do not have an estimate for this simply because the activity is
> > > >>>>> community-driven and it depends on the number of people willing
> to
> > > >>>>> contribute. With the current pace, I would hope to have an RC of
> > > Ignite
> > > >>>> 3.0
> > > >>>>> to be ready by the end of 2021. My gut feeling is that by moving
> > with
> > > >>>>> incremental changes, we will not be able to implement even half
> of
> > > the
> > > >>>>> wishlist by that time.
> > > >>>>> I doubt that releasing several major releases with breaking
> changes
> > > >> will
> > > >>>>> make Ignite users happy either because each upgrade will cost
> > Ignite
> > > >>>> users
> > > >>>>> money, so the fewer major versions we release, the better. Thus
> my
> > > wish
> > > >>>> to
> > > >>>>> include all breaking changes in one release.
> > > >>>>>
> > > >>>>> I'll be now quiet for a while, let's see what other community
> > members
> > > >>>>> think.
> > > >>>>>
> > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> [hidden email]
> > >:
> > > >>>>>
> > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > >>>>>> We don't want to follow the path of Netscape and lose all our
> > users
> > > >>>>>> by the time we have a working 3.0 [1]
> > > >>>>>>
> > > >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> > > what's
> > > >>>> the
> > > >>>>>> problem with a branch?
> > > >>>>>>
> > > >>>>>> 3. We should keep existing integration tests when possible.
> > > >>>>>> We have accumulated a lot of edge case knowledge over the years,
> > > >>>>>> it is not a good idea to send all of that down the drain.
> > > >>>>>> Yes, integration tests are slow, but they are the most valuable.
> > > >>>>>> I think we can move more stuff into nightly runs and have a fast
> > and
> > > >>>>> modern
> > > >>>>>> basic suite.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> > > than
> > > >>>>> most
> > > >>>>>> of us,
> > > >>>>>> can you please explain in more detail which particular feature,
> in
> > > >> your
> > > >>>>>> opinion,
> > > >>>>>> mandates this "start from scratch" approach?
> > > >>>>>> Is it really not possible at all to follow a less radical way?
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > >>>>>>
> > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > [hidden email]
> > > >
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hello, Alexey.
> > > >>>>>>>
> > > >>>>>>> I think that «rewriting from scratch» approach has a high risk
> to
> > > >>>> make
> > > >>>>>> new
> > > >>>>>>> features unusable.
> > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or
> bad
> > > >>>>>> features.
> > > >>>>>>> Nevertheless, it happen.
> > > >>>>>>>
> > > >>>>>>> I think we can avoid it with the Ignite3 and successors if we
> > will
> > > >>>> move
> > > >>>>>>> step by step without keeping backward compatibility
> > > >>>>>>> With the step by step approach, we can focus on each component
> > > >>>>>> separately.
> > > >>>>>>>
> > > >>>>>>>> What new features are we planning to implement for Ignite 2.x?
> > > >>>>>>>
> > > >>>>>>> We have many features that have to evolve.
> > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> breaking
> > > >>>>>> backward
> > > >>>>>>> compatibility.
> > > >>>>>>> We have many users who are happy with the 2.x with all it’s
> > issues.
> > > >>>>>>>
> > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> > > >>>>> написал(а):
> > > >>>>>>>>
> > > >>>>>>>> Alexey,
> > > >>>>>>>>
> > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > > >>>>>>> production-ready
> > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > >>>>>>> [hidden email]>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Nikolay,
> > > >>>>>>>>>
> > > >>>>>>>>> What new features are we planning to implement for Ignite
> 2.x?
> > I
> > > >>>>> think
> > > >>>>>>> once
> > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease
> > the
> > > >>>>>>> activity
> > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > development
> > > >>>>> will
> > > >>>>>> be
> > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > >>>>>>>>>
> > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > [hidden email]
> > > >>>>> :
> > > >>>>>>>>>
> > > >>>>>>>>>> To be clear:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> > > >>>>> (perhaps, a
> > > >>>>>>>>> new
> > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
> > Ignite
> > > >>>>> 3.0
> > > >>>>>>>>>> TeamCity project.
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for new Team City project.
> > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > >>>>>>>>>> -1 for new repo.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > >>>> [hidden email]
> > > >>>>>>
> > > >>>>>>>>>> написал(а):
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hello, Alexey.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > >>>>>>>>>>> Developing new features for 2 separate branches with the
> > > >>>> different
> > > >>>>>>> APIs
> > > >>>>>>>>>> and internal structure is overwhelming
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >>>>>>>>> [hidden email]>
> > > >>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Igniters,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the
> Ignite
> > > >>>> 3.0
> > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which
> > > imply
> > > >>>>>> major
> > > >>>>>>>>>>>> changes to the codebase (the change in replication
> protocol
> > > and
> > > >>>>>> thus
> > > >>>>>>>>>>>> transactions, change in binary format, updated
> metastorage,
> > > >>>> etc).
> > > >>>>>> We
> > > >>>>>>>>>> also
> > > >>>>>>>>>>>> planned significant changes in public APIs: configuration
> > > >>>> format
> > > >>>>>>>>> change,
> > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> > rework.
> > > >>>>> The
> > > >>>>>>>>>> wishlist
> > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> change
> > > the
> > > >>>>> old
> > > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces
> > of
> > > >>>>> code
> > > >>>>>>>>> that
> > > >>>>>>>>>> do
> > > >>>>>>>>>>>> not require significant rework. Personally, I would go
> with
> > > the
> > > >>>>>>> second
> > > >>>>>>>>>>>> option for the following reasons:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - We have a chance to shift the development paradigm in
> the
> > > >>>>> project
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> > baseline
> > > >>>> at
> > > >>>>>> the
> > > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> > > >>>> scenario,
> > > >>>>>>>>> thus
> > > >>>>>>>>>> we
> > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice
> > was
> > > >>>>> hard
> > > >>>>>> to
> > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > components
> > > >>>> and
> > > >>>>>>>>>> inability
> > > >>>>>>>>>>>> to instantiate components without an instance of
> > > KernalContext.
> > > >>>>> For
> > > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > > >>>>> primitives,
> > > >>>>>>>>>> such as
> > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > >>>> distributed
> > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > >>>>>>>>>>>> - We will significantly reduce the development cycle in
> the
> > > >>>>>> beginning
> > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time
> > > with
> > > >>>>>> empty
> > > >>>>>>>>>> TC;
> > > >>>>>>>>>>>> in the new approach developer will be able to run ALL
> tests
> > > >>>>> locally
> > > >>>>>>>>> in
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> matter of minutes)
> > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > integrating
> > > >>>> TC
> > > >>>>>>>>> build
> > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> > > >>>>>> integrated
> > > >>>>>>>>>> to PR
> > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > > >>>>>>>>>>>> - We will still have to re-write all tests, but only once.
> > If
> > > >>>> we
> > > >>>>>> try
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> modify the old codebase, we would need to modify all the
> > tests
> > > >>>>> for
> > > >>>>>>>>>> every
> > > >>>>>>>>>>>> major change (public API change, configuration change)
> > > >>>>>>>>>>>> - We will have fewer conflicts when working together. For
> > > >>>>> example,
> > > >>>>>> I
> > > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting
> > rid
> > > >>>> of
> > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> > example
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Technically, I would suggest creating a new repository for
> > > >>>> Ignite
> > > >>>>>> 3.0
> > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer
> to
> > > me)
> > > >>>>>> and a
> > > >>>>>>>>>> new
> > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> > > >>>> approach
> > > >>>>>> will
> > > >>>>>>>>>> give
> > > >>>>>>>>>>>> us more benefits than trying to make such major changes in
> > the
> > > >>>>>>>>> existing
> > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like
> > before
> > > >>>> to
> > > >>>>>>>>>> discuss
> > > >>>>>>>>>>>> this.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best regards,
> > > >>>> Andrey V. Mashenkov
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

[Valentin Kulichenko]

Ksenia, thanks for scheduling this on such short notice!

As for the original topic, I do support Alexey's idea. We're not going to
rewrite anything from scratch, as most of the components are going to be
moved as-is or with minimal modifications. However, the changes that are
proposed imply serious rework of the core parts of the code, which are not
properly decoupled from each other and from other parts. This makes the
incremental approach borderline impossible. Developing in a new repo,
however, addresses this concern. As a bonus, we can also refactor the code,
introduce better decoupling, get rid of kernel context, and develop unit
tests (finally!).

Basically, this proposal only affects the *process*, not the set of changes
we had discussed before. Ignite 3.0 is our unique chance to make things
right.

-Val

On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <[hidden email]>
wrote:

> Pavel, all the interesting points will be anyway published here in English
> (as the principal "if it's not on devlist it doesn't happened" is still
> relevant). This is just a quick call for a group of developers. Later we
> can do a separate presentation of idea and discussion in English as we did
> for the Ignite 3.0 draft of changes.
>
> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <[hidden email]>:
>
> > Kseniya,
> >
> > Thanks for scheduling this call.
> > Do you think we can switch to English if non-Russian speaking community
> > members decide to join?
> >
> > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> [hidden email]
> > >
> > wrote:
> >
> > > Let's do this community discussion open. Here's the link on zoom call
> in
> > > Russian for Friday 6 PM:
> > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > >
> > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <[hidden email]>:
> > >
> > > > Time works for me.
> > > >
> > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > [hidden email]
> > > >
> > > > написал(а):
> > > > >
> > > > > Nikolay,
> > > > >
> > > > > I am up for the call. I will try to explain my reasoning in greater
> > > > detail
> > > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> > work?
> > > > >
> > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <[hidden email]
> >:
> > > > >
> > > > >> Igniters, should we have a call for this topic?
> > > > >>
> > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]>
> > > > >> написал(а):
> > > > >>>
> > > > >>>> not intend to rewrite everything from scratch
> > > > >>>
> > > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > > > >>>> regardless of how we choose to proceed.
> > > > >>>
> > > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > > concerns.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > >> [hidden email]>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi, Igniters.
> > > > >>>>
> > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > restrictions
> > > > >> to
> > > > >>>> pull requests,
> > > > >>>> otherwise I see no difference for myself.
> > > > >>>> E.g. make static analysis (do we have?), compile, styles, and
> > > javadoc
> > > > >>>> checks mandatory.
> > > > >>>>
> > > > >>>> I think that relaxed requirements here will lead to bad product
> > > > quality.
> > > > >>>>
> > > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > > > >>>> During active development tests will be broken most of time, so,
> > > > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > > > >> featured
> > > > >>>> environment to run them and of course make test's code clear and
> > > avoid
> > > > >>>> bad/non-relevant ones.
> > > > >>>>
> > > > >>>> * I like bottom-up approach.
> > > > >>>> With it we could make a better framework. I mean clear component
> > > > >> lifecycle,
> > > > >>>> component wiring mechanics, general methods to approach core
> > > > components
> > > > >>>> such as exchange/communication
> > > > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > > > custom
> > > > >>>> callbacks for each component, interfaces like
> > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > > >>>> a pack of
> > > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > >>>> and so on in various unexpected places.
> > > > >>>> Hope, we will be able to port most of the good code to the new
> > > > framework
> > > > >>>> version.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > >>>> [hidden email]>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Nikolay, Pavel,
> > > > >>>>>
> > > > >>>>> Thanks for the feedback! First of all, I wanted to stress that
> I
> > do
> > > > not
> > > > >>>>> intend to rewrite everything from scratch (I never used this
> > > phrase).
> > > > >>>> There
> > > > >>>>> are significant parts of code that would be moved with minimal
> > > > >>>>> modifications. Second, I never said that we will get rid of the
> > old
> > > > >> tests
> > > > >>>>> codebase. Every single test from Ignite 2.x should be moved to
> > > > Ignite 3
> > > > >>>>> regardless of how we choose to proceed.
> > > > >>>>>
> > > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > > >>>>> implementation will be cheaper in many ways. Let me give you a
> > few
> > > > >>>> concrete
> > > > >>>>> examples:
> > > > >>>>>
> > > > >>>>>  - I think no one can object that we need a cleanly separated
> > > > >>>> persistence
> > > > >>>>>  layer for Ignite. There is a very raw draft IEP for this
> > already.
> > > On
> > > > >>>> the
> > > > >>>>>  other hand, I think we also can agree that we need a
> split-brain
> > > > >>>>> resistant
> > > > >>>>>  replication protocol for caches. There is also an IEP for
> this.
> > > > >>>> Neither
> > > > >>>>> of
> > > > >>>>>  the changes is a good fit for 2.x because they are likely to
> > > > >> introduce
> > > > >>>>>  breaking changes in the persistence layer, configuration and
> > > > >> behavior.
> > > > >>>>>  Additionally, these components are now tightly coupled, so
> there
> > > is
> > > > >> no
> > > > >>>>> way
> > > > >>>>>  these two changes can be implemented in parallel and then
> merged
> > > > >>>>> together
> > > > >>>>>  easily. So what we will end up with is having to implement
> these
> > > > >>>> changes
> > > > >>>>>  sequentially, fixing all existing tests twice, and essentially
> > > > >>>> throwing
> > > > >>>>>  away half of the work done because the other part of the
> change
> > is
> > > > >>>>>  re-implemented
> > > > >>>>>  - Similar example goes with getting rid of
> IgniteInternalFuture
> > > and
> > > > >>>>>  replacing it with CompletableFuture, and any other change that
> > > > >> touches
> > > > >>>>> the
> > > > >>>>>  asynchronous part of the code.
> > > > >>>>>
> > > > >>>>> Third, I do not see how this choice affects the UX of Ignite.
> The
> > > end
> > > > >>>> user
> > > > >>>>> experience must be fixed in the IEP regardless of the
> development
> > > > >> process
> > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > > > confirms
> > > > >>>>> that.
> > > > >>>>>
> > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > > >>>> reformulate,
> > > > >>>>> my point is that we might agree to have a single development
> > master
> > > > >>>> branch
> > > > >>>>> with 'disassembled' end-to-end functionality for some period of
> > > time
> > > > to
> > > > >>>>> speed up development, and re-assemble the core features after
> > > having
> > > > >>>>> submodules tested independently.
> > > > >>>>>
> > > > >>>>> Nikolay,
> > > > >>>>>> We have many features that have to evolve.
> > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> etc.
> > > > >>>>> This is not very specific. In the end, resources are limited
> and
> > we
> > > > >> will
> > > > >>>>> not be able to drive both tracks simultaneously, especially
> > after a
> > > > >>>> couple
> > > > >>>>> of features having been implemented for Ignite 3.0. If there
> are
> > > > indeed
> > > > >>>>> some major changes that we want to do in Ignite 2.x instead of
> > > > putting
> > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of
> any,
> > > > >> that's
> > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > >>>>>
> > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > breaking
> > > > >>>> backward
> > > > >>>>> compatibility.
> > > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > > issues.
> > > > >>>>> These changes will be covered by end-to-end tests and migrated
> to
> > > > >> Ignite
> > > > >>>>> 3.0, so I see no issues here.
> > > > >>>>>
> > > > >>>>> Finally, Anton & Nikolay
> > > > >>>>> I do not have an estimate for this simply because the activity
> is
> > > > >>>>> community-driven and it depends on the number of people willing
> > to
> > > > >>>>> contribute. With the current pace, I would hope to have an RC
> of
> > > > Ignite
> > > > >>>> 3.0
> > > > >>>>> to be ready by the end of 2021. My gut feeling is that by
> moving
> > > with
> > > > >>>>> incremental changes, we will not be able to implement even half
> > of
> > > > the
> > > > >>>>> wishlist by that time.
> > > > >>>>> I doubt that releasing several major releases with breaking
> > changes
> > > > >> will
> > > > >>>>> make Ignite users happy either because each upgrade will cost
> > > Ignite
> > > > >>>> users
> > > > >>>>> money, so the fewer major versions we release, the better. Thus
> > my
> > > > wish
> > > > >>>> to
> > > > >>>>> include all breaking changes in one release.
> > > > >>>>>
> > > > >>>>> I'll be now quiet for a while, let's see what other community
> > > members
> > > > >>>>> think.
> > > > >>>>>
> > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > [hidden email]
> > > >:
> > > > >>>>>
> > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > >>>>>> We don't want to follow the path of Netscape and lose all our
> > > users
> > > > >>>>>> by the time we have a working 3.0 [1]
> > > > >>>>>>
> > > > >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> > > > what's
> > > > >>>> the
> > > > >>>>>> problem with a branch?
> > > > >>>>>>
> > > > >>>>>> 3. We should keep existing integration tests when possible.
> > > > >>>>>> We have accumulated a lot of edge case knowledge over the
> years,
> > > > >>>>>> it is not a good idea to send all of that down the drain.
> > > > >>>>>> Yes, integration tests are slow, but they are the most
> valuable.
> > > > >>>>>> I think we can move more stuff into nightly runs and have a
> fast
> > > and
> > > > >>>>> modern
> > > > >>>>>> basic suite.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Alexey, you are much more familiar with the Ignite core
> codebase
> > > > than
> > > > >>>>> most
> > > > >>>>>> of us,
> > > > >>>>>> can you please explain in more detail which particular
> feature,
> > in
> > > > >> your
> > > > >>>>>> opinion,
> > > > >>>>>> mandates this "start from scratch" approach?
> > > > >>>>>> Is it really not possible at all to follow a less radical way?
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> [1]
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > >>>>>>
> > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > [hidden email]
> > > > >
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hello, Alexey.
> > > > >>>>>>>
> > > > >>>>>>> I think that «rewriting from scratch» approach has a high
> risk
> > to
> > > > >>>> make
> > > > >>>>>> new
> > > > >>>>>>> features unusable.
> > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or
> > bad
> > > > >>>>>> features.
> > > > >>>>>>> Nevertheless, it happen.
> > > > >>>>>>>
> > > > >>>>>>> I think we can avoid it with the Ignite3 and successors if we
> > > will
> > > > >>>> move
> > > > >>>>>>> step by step without keeping backward compatibility
> > > > >>>>>>> With the step by step approach, we can focus on each
> component
> > > > >>>>>> separately.
> > > > >>>>>>>
> > > > >>>>>>>> What new features are we planning to implement for Ignite
> 2.x?
> > > > >>>>>>>
> > > > >>>>>>> We have many features that have to evolve.
> > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> etc.
> > > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> > breaking
> > > > >>>>>> backward
> > > > >>>>>>> compatibility.
> > > > >>>>>>> We have many users who are happy with the 2.x with all it’s
> > > issues.
> > > > >>>>>>>
> > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]>
> > > > >>>>> написал(а):
> > > > >>>>>>>>
> > > > >>>>>>>> Alexey,
> > > > >>>>>>>>
> > > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > > > >>>>>>> production-ready
> > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > >>>>>>>>
> > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > >>>>>>> [hidden email]>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Nikolay,
> > > > >>>>>>>>>
> > > > >>>>>>>>> What new features are we planning to implement for Ignite
> > 2.x?
> > > I
> > > > >>>>> think
> > > > >>>>>>> once
> > > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually
> cease
> > > the
> > > > >>>>>>> activity
> > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > > development
> > > > >>>>> will
> > > > >>>>>> be
> > > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > [hidden email]
> > > > >>>>> :
> > > > >>>>>>>>>
> > > > >>>>>>>>>> To be clear:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> > > > >>>>> (perhaps, a
> > > > >>>>>>>>> new
> > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
> > > Ignite
> > > > >>>>> 3.0
> > > > >>>>>>>>>> TeamCity project.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> +1 for new Team City project.
> > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > >>>>>>>>>> -1 for new repo.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > >>>> [hidden email]
> > > > >>>>>>
> > > > >>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hello, Alexey.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > >>>>>>>>>>> Developing new features for 2 separate branches with the
> > > > >>>> different
> > > > >>>>>>> APIs
> > > > >>>>>>>>>> and internal structure is overwhelming
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with
> new
> > > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > >>>>>>>>> [hidden email]>
> > > > >>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Igniters,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the
> > Ignite
> > > > >>>> 3.0
> > > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0
> which
> > > > imply
> > > > >>>>>> major
> > > > >>>>>>>>>>>> changes to the codebase (the change in replication
> > protocol
> > > > and
> > > > >>>>>> thus
> > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > metastorage,
> > > > >>>> etc).
> > > > >>>>>> We
> > > > >>>>>>>>>> also
> > > > >>>>>>>>>>>> planned significant changes in public APIs:
> configuration
> > > > >>>> format
> > > > >>>>>>>>> change,
> > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> > > rework.
> > > > >>>>> The
> > > > >>>>>>>>>> wishlist
> > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> > change
> > > > the
> > > > >>>>> old
> > > > >>>>>>>>>>>> codebase, or start with a new baseline and move old
> pieces
> > > of
> > > > >>>>> code
> > > > >>>>>>>>> that
> > > > >>>>>>>>>> do
> > > > >>>>>>>>>>>> not require significant rework. Personally, I would go
> > with
> > > > the
> > > > >>>>>>> second
> > > > >>>>>>>>>>>> option for the following reasons:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> - We have a chance to shift the development paradigm in
> > the
> > > > >>>>> project
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> > > baseline
> > > > >>>> at
> > > > >>>>>> the
> > > > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> > > > >>>> scenario,
> > > > >>>>>>>>> thus
> > > > >>>>>>>>>> we
> > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> practice
> > > was
> > > > >>>>> hard
> > > > >>>>>> to
> > > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > > components
> > > > >>>> and
> > > > >>>>>>>>>> inability
> > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > KernalContext.
> > > > >>>>> For
> > > > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > > > >>>>> primitives,
> > > > >>>>>>>>>> such as
> > > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > > >>>> distributed
> > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > >>>>>>>>>>>> - We will significantly reduce the development cycle in
> > the
> > > > >>>>>> beginning
> > > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical
> time
> > > > with
> > > > >>>>>> empty
> > > > >>>>>>>>>> TC;
> > > > >>>>>>>>>>>> in the new approach developer will be able to run ALL
> > tests
> > > > >>>>> locally
> > > > >>>>>>>>> in
> > > > >>>>>>>>>> a
> > > > >>>>>>>>>>>> matter of minutes)
> > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > > integrating
> > > > >>>> TC
> > > > >>>>>>>>> build
> > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> currently
> > > > >>>>>> integrated
> > > > >>>>>>>>>> to PR
> > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > > > >>>>>>>>>>>> - We will still have to re-write all tests, but only
> once.
> > > If
> > > > >>>> we
> > > > >>>>>> try
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> modify the old codebase, we would need to modify all the
> > > tests
> > > > >>>>> for
> > > > >>>>>>>>>> every
> > > > >>>>>>>>>>>> major change (public API change, configuration change)
> > > > >>>>>>>>>>>> - We will have fewer conflicts when working together.
> For
> > > > >>>>> example,
> > > > >>>>>> I
> > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of
> getting
> > > rid
> > > > >>>> of
> > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> > > example
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Technically, I would suggest creating a new repository
> for
> > > > >>>> Ignite
> > > > >>>>>> 3.0
> > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer
> > to
> > > > me)
> > > > >>>>>> and a
> > > > >>>>>>>>>> new
> > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> > > > >>>> approach
> > > > >>>>>> will
> > > > >>>>>>>>>> give
> > > > >>>>>>>>>>>> us more benefits than trying to make such major changes
> in
> > > the
> > > > >>>>>>>>> existing
> > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like
> > > before
> > > > >>>> to
> > > > >>>>>>>>>> discuss
> > > > >>>>>>>>>>>> this.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Best regards,
> > > > >>>> Andrey V. Mashenkov
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

[Anton Vinogradov-2]

Folks,

Should we perform cleanup work before (r)evolutional changes?
My huge proposal is to get rid of things which we don't need anyway
- local caches,
- strange tx modes,
- code overcomplexity because of RollingUpgrade feature never attended at
AI,
- etc,
before choosing the way.

On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
[hidden email]> wrote:

> Ksenia, thanks for scheduling this on such short notice!
>
> As for the original topic, I do support Alexey's idea. We're not going to
> rewrite anything from scratch, as most of the components are going to be
> moved as-is or with minimal modifications. However, the changes that are
> proposed imply serious rework of the core parts of the code, which are not
> properly decoupled from each other and from other parts. This makes the
> incremental approach borderline impossible. Developing in a new repo,
> however, addresses this concern. As a bonus, we can also refactor the code,
> introduce better decoupling, get rid of kernel context, and develop unit
> tests (finally!).
>
> Basically, this proposal only affects the *process*, not the set of changes
> we had discussed before. Ignite 3.0 is our unique chance to make things
> right.
>
> -Val
>
> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <[hidden email]
> >
> wrote:
>
> > Pavel, all the interesting points will be anyway published here in
> English
> > (as the principal "if it's not on devlist it doesn't happened" is still
> > relevant). This is just a quick call for a group of developers. Later we
> > can do a separate presentation of idea and discussion in English as we
> did
> > for the Ignite 3.0 draft of changes.
> >
> > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <[hidden email]>:
> >
> > > Kseniya,
> > >
> > > Thanks for scheduling this call.
> > > Do you think we can switch to English if non-Russian speaking community
> > > members decide to join?
> > >
> > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > Let's do this community discussion open. Here's the link on zoom call
> > in
> > > > Russian for Friday 6 PM:
> > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > >
> > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <[hidden email]>:
> > > >
> > > > > Time works for me.
> > > > >
> > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > [hidden email]
> > > > >
> > > > > написал(а):
> > > > > >
> > > > > > Nikolay,
> > > > > >
> > > > > > I am up for the call. I will try to explain my reasoning in
> greater
> > > > > detail
> > > > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> > > work?
> > > > > >
> > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> [hidden email]
> > >:
> > > > > >
> > > > > >> Igniters, should we have a call for this topic?
> > > > > >>
> > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <[hidden email]
> >
> > > > > >> написал(а):
> > > > > >>>
> > > > > >>>> not intend to rewrite everything from scratch
> > > > > >>>
> > > > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > > > > >>>> regardless of how we choose to proceed.
> > > > > >>>
> > > > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > > > concerns.
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > >> [hidden email]>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hi, Igniters.
> > > > > >>>>
> > > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > > restrictions
> > > > > >> to
> > > > > >>>> pull requests,
> > > > > >>>> otherwise I see no difference for myself.
> > > > > >>>> E.g. make static analysis (do we have?), compile, styles, and
> > > > javadoc
> > > > > >>>> checks mandatory.
> > > > > >>>>
> > > > > >>>> I think that relaxed requirements here will lead to bad
> product
> > > > > quality.
> > > > > >>>>
> > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> somehow.
> > > > > >>>> During active development tests will be broken most of time,
> so,
> > > > > >>>> I'd port them e.g. suite-by-suite once we will have a stable
> and
> > > > > >> featured
> > > > > >>>> environment to run them and of course make test's code clear
> and
> > > > avoid
> > > > > >>>> bad/non-relevant ones.
> > > > > >>>>
> > > > > >>>> * I like bottom-up approach.
> > > > > >>>> With it we could make a better framework. I mean clear
> component
> > > > > >> lifecycle,
> > > > > >>>> component wiring mechanics, general methods to approach core
> > > > > components
> > > > > >>>> such as exchange/communication
> > > > > >>>> to avoid code mess like we have in ExchangeFuture with all
> these
> > > > > custom
> > > > > >>>> callbacks for each component, interfaces like
> > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > > > >>>> a pack of
> > > > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > >>>> and so on in various unexpected places.
> > > > > >>>> Hope, we will be able to port most of the good code to the new
> > > > > framework
> > > > > >>>> version.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > >>>> [hidden email]>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Nikolay, Pavel,
> > > > > >>>>>
> > > > > >>>>> Thanks for the feedback! First of all, I wanted to stress
> that
> > I
> > > do
> > > > > not
> > > > > >>>>> intend to rewrite everything from scratch (I never used this
> > > > phrase).
> > > > > >>>> There
> > > > > >>>>> are significant parts of code that would be moved with
> minimal
> > > > > >>>>> modifications. Second, I never said that we will get rid of
> the
> > > old
> > > > > >> tests
> > > > > >>>>> codebase. Every single test from Ignite 2.x should be moved
> to
> > > > > Ignite 3
> > > > > >>>>> regardless of how we choose to proceed.
> > > > > >>>>>
> > > > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > > > >>>>> implementation will be cheaper in many ways. Let me give you
> a
> > > few
> > > > > >>>> concrete
> > > > > >>>>> examples:
> > > > > >>>>>
> > > > > >>>>>  - I think no one can object that we need a cleanly separated
> > > > > >>>> persistence
> > > > > >>>>>  layer for Ignite. There is a very raw draft IEP for this
> > > already.
> > > > On
> > > > > >>>> the
> > > > > >>>>>  other hand, I think we also can agree that we need a
> > split-brain
> > > > > >>>>> resistant
> > > > > >>>>>  replication protocol for caches. There is also an IEP for
> > this.
> > > > > >>>> Neither
> > > > > >>>>> of
> > > > > >>>>>  the changes is a good fit for 2.x because they are likely to
> > > > > >> introduce
> > > > > >>>>>  breaking changes in the persistence layer, configuration and
> > > > > >> behavior.
> > > > > >>>>>  Additionally, these components are now tightly coupled, so
> > there
> > > > is
> > > > > >> no
> > > > > >>>>> way
> > > > > >>>>>  these two changes can be implemented in parallel and then
> > merged
> > > > > >>>>> together
> > > > > >>>>>  easily. So what we will end up with is having to implement
> > these
> > > > > >>>> changes
> > > > > >>>>>  sequentially, fixing all existing tests twice, and
> essentially
> > > > > >>>> throwing
> > > > > >>>>>  away half of the work done because the other part of the
> > change
> > > is
> > > > > >>>>>  re-implemented
> > > > > >>>>>  - Similar example goes with getting rid of
> > IgniteInternalFuture
> > > > and
> > > > > >>>>>  replacing it with CompletableFuture, and any other change
> that
> > > > > >> touches
> > > > > >>>>> the
> > > > > >>>>>  asynchronous part of the code.
> > > > > >>>>>
> > > > > >>>>> Third, I do not see how this choice affects the UX of Ignite.
> > The
> > > > end
> > > > > >>>> user
> > > > > >>>>> experience must be fixed in the IEP regardless of the
> > development
> > > > > >> process
> > > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x
> just
> > > > > confirms
> > > > > >>>>> that.
> > > > > >>>>>
> > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > > > >>>> reformulate,
> > > > > >>>>> my point is that we might agree to have a single development
> > > master
> > > > > >>>> branch
> > > > > >>>>> with 'disassembled' end-to-end functionality for some period
> of
> > > > time
> > > > > to
> > > > > >>>>> speed up development, and re-assemble the core features after
> > > > having
> > > > > >>>>> submodules tested independently.
> > > > > >>>>>
> > > > > >>>>> Nikolay,
> > > > > >>>>>> We have many features that have to evolve.
> > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> > etc.
> > > > > >>>>> This is not very specific. In the end, resources are limited
> > and
> > > we
> > > > > >> will
> > > > > >>>>> not be able to drive both tracks simultaneously, especially
> > > after a
> > > > > >>>> couple
> > > > > >>>>> of features having been implemented for Ignite 3.0. If there
> > are
> > > > > indeed
> > > > > >>>>> some major changes that we want to do in Ignite 2.x instead
> of
> > > > > putting
> > > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of
> > any,
> > > > > >> that's
> > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > >>>>>
> > > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > > breaking
> > > > > >>>> backward
> > > > > >>>>> compatibility.
> > > > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > > > issues.
> > > > > >>>>> These changes will be covered by end-to-end tests and
> migrated
> > to
> > > > > >> Ignite
> > > > > >>>>> 3.0, so I see no issues here.
> > > > > >>>>>
> > > > > >>>>> Finally, Anton & Nikolay
> > > > > >>>>> I do not have an estimate for this simply because the
> activity
> > is
> > > > > >>>>> community-driven and it depends on the number of people
> willing
> > > to
> > > > > >>>>> contribute. With the current pace, I would hope to have an RC
> > of
> > > > > Ignite
> > > > > >>>> 3.0
> > > > > >>>>> to be ready by the end of 2021. My gut feeling is that by
> > moving
> > > > with
> > > > > >>>>> incremental changes, we will not be able to implement even
> half
> > > of
> > > > > the
> > > > > >>>>> wishlist by that time.
> > > > > >>>>> I doubt that releasing several major releases with breaking
> > > changes
> > > > > >> will
> > > > > >>>>> make Ignite users happy either because each upgrade will cost
> > > > Ignite
> > > > > >>>> users
> > > > > >>>>> money, so the fewer major versions we release, the better.
> Thus
> > > my
> > > > > wish
> > > > > >>>> to
> > > > > >>>>> include all breaking changes in one release.
> > > > > >>>>>
> > > > > >>>>> I'll be now quiet for a while, let's see what other community
> > > > members
> > > > > >>>>> think.
> > > > > >>>>>
> > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > [hidden email]
> > > > >:
> > > > > >>>>>
> > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > >>>>>> We don't want to follow the path of Netscape and lose all
> our
> > > > users
> > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > >>>>>>
> > > > > >>>>>> 2. Not sure about new repo - seems like some pain and no
> gain,
> > > > > what's
> > > > > >>>> the
> > > > > >>>>>> problem with a branch?
> > > > > >>>>>>
> > > > > >>>>>> 3. We should keep existing integration tests when possible.
> > > > > >>>>>> We have accumulated a lot of edge case knowledge over the
> > years,
> > > > > >>>>>> it is not a good idea to send all of that down the drain.
> > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > valuable.
> > > > > >>>>>> I think we can move more stuff into nightly runs and have a
> > fast
> > > > and
> > > > > >>>>> modern
> > > > > >>>>>> basic suite.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> Alexey, you are much more familiar with the Ignite core
> > codebase
> > > > > than
> > > > > >>>>> most
> > > > > >>>>>> of us,
> > > > > >>>>>> can you please explain in more detail which particular
> > feature,
> > > in
> > > > > >> your
> > > > > >>>>>> opinion,
> > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > >>>>>> Is it really not possible at all to follow a less radical
> way?
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> [1]
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > [hidden email]
> > > > > >
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hello, Alexey.
> > > > > >>>>>>>
> > > > > >>>>>>> I think that «rewriting from scratch» approach has a high
> > risk
> > > to
> > > > > >>>> make
> > > > > >>>>>> new
> > > > > >>>>>>> features unusable.
> > > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX
> or
> > > bad
> > > > > >>>>>> features.
> > > > > >>>>>>> Nevertheless, it happen.
> > > > > >>>>>>>
> > > > > >>>>>>> I think we can avoid it with the Ignite3 and successors if
> we
> > > > will
> > > > > >>>> move
> > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > >>>>>>> With the step by step approach, we can focus on each
> > component
> > > > > >>>>>> separately.
> > > > > >>>>>>>
> > > > > >>>>>>>> What new features are we planning to implement for Ignite
> > 2.x?
> > > > > >>>>>>>
> > > > > >>>>>>> We have many features that have to evolve.
> > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> > etc.
> > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> > > breaking
> > > > > >>>>>> backward
> > > > > >>>>>>> compatibility.
> > > > > >>>>>>> We have many users who are happy with the 2.x with all it’s
> > > > issues.
> > > > > >>>>>>>
> > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <[hidden email]
> >
> > > > > >>>>> написал(а):
> > > > > >>>>>>>>
> > > > > >>>>>>>> Alexey,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > > > > >>>>>>> production-ready
> > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > >>>>>>> [hidden email]>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Nikolay,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> What new features are we planning to implement for Ignite
> > > 2.x?
> > > > I
> > > > > >>>>> think
> > > > > >>>>>>> once
> > > > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually
> > cease
> > > > the
> > > > > >>>>>>> activity
> > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > > > development
> > > > > >>>>> will
> > > > > >>>>>> be
> > > > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > [hidden email]
> > > > > >>>>> :
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> To be clear:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> I would suggest creating a new repository for Ignite
> 3.0
> > > > > >>>>> (perhaps, a
> > > > > >>>>>>>>> new
> > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a
> new
> > > > Ignite
> > > > > >>>>> 3.0
> > > > > >>>>>>>>>> TeamCity project.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > >>>>>>>>>> -1 for new repo.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > >>>> [hidden email]
> > > > > >>>>>>
> > > > > >>>>>>>>>> написал(а):
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > >>>>>>>>>>> Developing new features for 2 separate branches with
> the
> > > > > >>>> different
> > > > > >>>>>>> APIs
> > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with
> > new
> > > > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > >>>>>>>>> [hidden email]>
> > > > > >>>>>>>>>> написал(а):
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Igniters,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the
> > > Ignite
> > > > > >>>> 3.0
> > > > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0
> > which
> > > > > imply
> > > > > >>>>>> major
> > > > > >>>>>>>>>>>> changes to the codebase (the change in replication
> > > protocol
> > > > > and
> > > > > >>>>>> thus
> > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > metastorage,
> > > > > >>>> etc).
> > > > > >>>>>> We
> > > > > >>>>>>>>>> also
> > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > configuration
> > > > > >>>> format
> > > > > >>>>>>>>> change,
> > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> > > > rework.
> > > > > >>>>> The
> > > > > >>>>>>>>>> wishlist
> > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> > > change
> > > > > the
> > > > > >>>>> old
> > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move old
> > pieces
> > > > of
> > > > > >>>>> code
> > > > > >>>>>>>>> that
> > > > > >>>>>>>>>> do
> > > > > >>>>>>>>>>>> not require significant rework. Personally, I would go
> > > with
> > > > > the
> > > > > >>>>>>> second
> > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> - We have a chance to shift the development paradigm
> in
> > > the
> > > > > >>>>> project
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> > > > baseline
> > > > > >>>> at
> > > > > >>>>>> the
> > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> end-to-end
> > > > > >>>> scenario,
> > > > > >>>>>>>>> thus
> > > > > >>>>>>>>>> we
> > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > practice
> > > > was
> > > > > >>>>> hard
> > > > > >>>>>> to
> > > > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > > > components
> > > > > >>>> and
> > > > > >>>>>>>>>> inability
> > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > KernalContext.
> > > > > >>>>> For
> > > > > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > > > > >>>>> primitives,
> > > > > >>>>>>>>>> such as
> > > > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > > > >>>> distributed
> > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > >>>>>>>>>>>> - We will significantly reduce the development cycle
> in
> > > the
> > > > > >>>>>> beginning
> > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical
> > time
> > > > > with
> > > > > >>>>>> empty
> > > > > >>>>>>>>>> TC;
> > > > > >>>>>>>>>>>> in the new approach developer will be able to run ALL
> > > tests
> > > > > >>>>> locally
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>> a
> > > > > >>>>>>>>>>>> matter of minutes)
> > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > > > integrating
> > > > > >>>> TC
> > > > > >>>>>>>>> build
> > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > currently
> > > > > >>>>>> integrated
> > > > > >>>>>>>>>> to PR
> > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but only
> > once.
> > > > If
> > > > > >>>> we
> > > > > >>>>>> try
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify all
> the
> > > > tests
> > > > > >>>>> for
> > > > > >>>>>>>>>> every
> > > > > >>>>>>>>>>>> major change (public API change, configuration change)
> > > > > >>>>>>>>>>>> - We will have fewer conflicts when working together.
> > For
> > > > > >>>>> example,
> > > > > >>>>>> I
> > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of
> > getting
> > > > rid
> > > > > >>>> of
> > > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> > > > example
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Technically, I would suggest creating a new repository
> > for
> > > > > >>>> Ignite
> > > > > >>>>>> 3.0
> > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks
> nicer
> > > to
> > > > > me)
> > > > > >>>>>> and a
> > > > > >>>>>>>>>> new
> > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that
> this
> > > > > >>>> approach
> > > > > >>>>>> will
> > > > > >>>>>>>>>> give
> > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> changes
> > in
> > > > the
> > > > > >>>>>>>>> existing
> > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat
> like
> > > > before
> > > > > >>>> to
> > > > > >>>>>>>>>> discuss
> > > > > >>>>>>>>>>>> this.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> WDYT?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> --
> > > > > >>>> Best regards,
> > > > > >>>> Andrey V. Mashenkov
> > > > > >>>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>