[DISCUSSION] The process of creating an Ignite Enhancement Proposal

[Mmuzaf]

Igniters,

I think, our community have accumulated enough experience with the process of
Ignite Enhancement Proposal (IEP) of introducing the major changes
into the Apache
Ignite. Now we have to take one step forward and make every major change and\or
improvement clear not only for community developers but also for the
Apache Ignite
users too.

Currently, I've seen two different strategies with creating IEPs:
 - Single IEP per single major improvement;
 - Single IEP per unit of functionality (group a few major improvements);

Both of them have different advantages and disadvantages.


Let's remove the ambiguity and build a convenient working process. For example,
we may consider the best practices used in other open-source projects [2] [3].

I propose to (and also propose to update the corresponding wiki page [1] and our
community processes):
 - use to "Single IEP per single major improvement" approach;
 - do not group the major enhancements into the single IEP;
 - avoid inaccurate proposal names (e.g. optimization, improvement,
stabilization of etc.);
 - for any public API changes create the IEP;
 - add a `compatibility\mirgration` section to IEP Template;
 - add a `public API changes` section to the IEP Template;
 - link each major release note with the corresponding IEP page (users
will have a
   better understanding of each feature).

 Igniters, please, share your thoughts.



[1] https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Enhancement+Proposal
[2] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
[3] https://cwiki.apache.org/confluence/display/SAMZA/Samza+Enhancement+Proposal

[Vladimir Ozerov]

Maxim,

I am not quite understand what is the problem with "many major enhancements
per IEP" and terms such as "optimization" or so. The very goal of initial
IEP process was to accumulate global ideas, so that one may quickly
understand potentially hot areas around the product. This is about
informing community, not about planning and linking to release notes.

Next, I fully agree that it is very good to have isolated IEP describing a
single major change. However, please note that a lot of IEPs start as
research activities. During this time we do not know what correct solution
is, how many major changes would be needed, how many components will be
affected, and how many tickets will be filed. All of this can change
drastically during IEP lifetime. Which looks perfectly fine for me - we
know what to improve, but do not know yet how exactly. If it is about real
implementation or planning you can always create umbrella ticket or so.
Moreover, it is also OK for IEPs to span multiple releases, e.g. this is
how we do this with TDE - huge change, multiple phases, multiple releases.

About names - again, ideally they should be concrete and non ambiguous. But
currently most of immediate product goals are around stabilization and
performance. This is really where we are. So if I, for example,
investigated a set of potential optimizations in area X, why can't I name
it "Optimize X"? If someone else will find more optimizations in the same
are while the first IEP is still active, he can join this IEP. If he find
them after first IEP is completed, then he can name it "Optimize X part 2",
while "Optimize X" will be in archive directory. So who really cares?

I do not see how enforcing any strict rules could help us here.

+1 for the rest.

On Wed, Nov 7, 2018 at 7:36 PM Maxim Muzafarov <[hidden email]> wrote:

> Igniters,
>
> I think, our community have accumulated enough experience with the process
> of
> Ignite Enhancement Proposal (IEP) of introducing the major changes
> into the Apache
> Ignite. Now we have to take one step forward and make every major change
> and\or
> improvement clear not only for community developers but also for the
> Apache Ignite
> users too.
>
> Currently, I've seen two different strategies with creating IEPs:
>  - Single IEP per single major improvement;
>  - Single IEP per unit of functionality (group a few major improvements);
>
> Both of them have different advantages and disadvantages.
>
>
> Let's remove the ambiguity and build a convenient working process. For
> example,
> we may consider the best practices used in other open-source projects [2]
> [3].
>
> I propose to (and also propose to update the corresponding wiki page [1]
> and our
> community processes):
>  - use to "Single IEP per single major improvement" approach;
>  - do not group the major enhancements into the single IEP;
>  - avoid inaccurate proposal names (e.g. optimization, improvement,
> stabilization of etc.);
>  - for any public API changes create the IEP;
>  - add a `compatibility\mirgration` section to IEP Template;
>  - add a `public API changes` section to the IEP Template;
>  - link each major release note with the corresponding IEP page (users
> will have a
>    better understanding of each feature).
>
>  Igniters, please, share your thoughts.
>
>
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Enhancement+Proposal
> [2]
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> [3]
> https://cwiki.apache.org/confluence/display/SAMZA/Samza+Enhancement+Proposal
>

[Mmuzaf]

Vladimir,

>  If someone else will find more optimizations in the same are while the first IEP is still active, he can join this IEP.

You fit the right problem case. If someone finds a new optimization
opportunity (it's totally normal) he joins to an active `IEP
Optimization X`. Simultaneously with the need to describe the design
of current major improvement united with the others major improvements
described by IEP the whole page dramatically increases. The whole
complexity becomes very big (multiple designs). This is what I worry
about.

My proposal is to choose between different strategies and not to allow
describing multiple major feature designs on a single page. Assume
these features are completely different.
So, what we have:

`Single IEP per unit of functionality`
 - IEP-100: Optimization X
   | - Change 1
   | - Change 2
   | - Change 3

or

`Single IEP per single major improvement`
 - IEP-100: Change 1
 - IEP-101: Change 2
 - IEP-102: Change 3

Personally, I like the second case.

> So if I, for example, investigated a set of potential optimizations in area X, why can't I name it "Optimize X"?

You can of course. It's not the strict rule.
If we have a good named isolated IEPs we implicitly help developers
not join some active IEP with a broad scope but create their single
short design document with accurate major improvement and improvement
design.
On Wed, 7 Nov 2018 at 20:40, Vladimir Ozerov <[hidden email]> wrote:

>
> Maxim,
>
> I am not quite understand what is the problem with "many major enhancements
> per IEP" and terms such as "optimization" or so. The very goal of initial
> IEP process was to accumulate global ideas, so that one may quickly
> understand potentially hot areas around the product. This is about
> informing community, not about planning and linking to release notes.
>
> Next, I fully agree that it is very good to have isolated IEP describing a
> single major change. However, please note that a lot of IEPs start as
> research activities. During this time we do not know what correct solution
> is, how many major changes would be needed, how many components will be
> affected, and how many tickets will be filed. All of this can change
> drastically during IEP lifetime. Which looks perfectly fine for me - we
> know what to improve, but do not know yet how exactly. If it is about real
> implementation or planning you can always create umbrella ticket or so.
> Moreover, it is also OK for IEPs to span multiple releases, e.g. this is
> how we do this with TDE - huge change, multiple phases, multiple releases.
>
> About names - again, ideally they should be concrete and non ambiguous. But
> currently most of immediate product goals are around stabilization and
> performance. This is really where we are. So if I, for example,
> investigated a set of potential optimizations in area X, why can't I name
> it "Optimize X"? If someone else will find more optimizations in the same
> are while the first IEP is still active, he can join this IEP. If he find
> them after first IEP is completed, then he can name it "Optimize X part 2",
> while "Optimize X" will be in archive directory. So who really cares?
>
> I do not see how enforcing any strict rules could help us here.
>
> +1 for the rest.
>
> On Wed, Nov 7, 2018 at 7:36 PM Maxim Muzafarov <[hidden email]> wrote:
>
> > Igniters,
> >
> > I think, our community have accumulated enough experience with the process
> > of
> > Ignite Enhancement Proposal (IEP) of introducing the major changes
> > into the Apache
> > Ignite. Now we have to take one step forward and make every major change
> > and\or
> > improvement clear not only for community developers but also for the
> > Apache Ignite
> > users too.
> >
> > Currently, I've seen two different strategies with creating IEPs:
> >  - Single IEP per single major improvement;
> >  - Single IEP per unit of functionality (group a few major improvements);
> >
> > Both of them have different advantages and disadvantages.
> >
> >
> > Let's remove the ambiguity and build a convenient working process. For
> > example,
> > we may consider the best practices used in other open-source projects [2]
> > [3].
> >
> > I propose to (and also propose to update the corresponding wiki page [1]
> > and our
> > community processes):
> >  - use to "Single IEP per single major improvement" approach;
> >  - do not group the major enhancements into the single IEP;
> >  - avoid inaccurate proposal names (e.g. optimization, improvement,
> > stabilization of etc.);
> >  - for any public API changes create the IEP;
> >  - add a `compatibility\mirgration` section to IEP Template;
> >  - add a `public API changes` section to the IEP Template;
> >  - link each major release note with the corresponding IEP page (users
> > will have a
> >    better understanding of each feature).
> >
> >  Igniters, please, share your thoughts.
> >
> >
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Enhancement+Proposal
> > [2]
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > [3]
> > https://cwiki.apache.org/confluence/display/SAMZA/Samza+Enhancement+Proposal
> >

[Ivan Pavlukhin]

Hi Maxim,

Single IEP per a major change looks desirable for me. But
I have doubts that it is always feasible.

Regarding naming. Could you please provide a couple of
examples of inaccurate names and how they might have
been improved?


чт, 8 нояб. 2018 г. в 21:19, Maxim Muzafarov <[hidden email]>:

> Vladimir,
>
> >  If someone else will find more optimizations in the same are while the
> first IEP is still active, he can join this IEP.
>
> You fit the right problem case. If someone finds a new optimization
> opportunity (it's totally normal) he joins to an active `IEP
> Optimization X`. Simultaneously with the need to describe the design
> of current major improvement united with the others major improvements
> described by IEP the whole page dramatically increases. The whole
> complexity becomes very big (multiple designs). This is what I worry
> about.
>
> My proposal is to choose between different strategies and not to allow
> describing multiple major feature designs on a single page. Assume
> these features are completely different.
> So, what we have:
>
> `Single IEP per unit of functionality`
>  - IEP-100: Optimization X
>    | - Change 1
>    | - Change 2
>    | - Change 3
>
> or
>
> `Single IEP per single major improvement`
>  - IEP-100: Change 1
>  - IEP-101: Change 2
>  - IEP-102: Change 3
>
> Personally, I like the second case.
>
> > So if I, for example, investigated a set of potential optimizations in
> area X, why can't I name it "Optimize X"?
>
> You can of course. It's not the strict rule.
> If we have a good named isolated IEPs we implicitly help developers
> not join some active IEP with a broad scope but create their single
> short design document with accurate major improvement and improvement
> design.
> On Wed, 7 Nov 2018 at 20:40, Vladimir Ozerov <[hidden email]> wrote:
> >
> > Maxim,
> >
> > I am not quite understand what is the problem with "many major
> enhancements
> > per IEP" and terms such as "optimization" or so. The very goal of initial
> > IEP process was to accumulate global ideas, so that one may quickly
> > understand potentially hot areas around the product. This is about
> > informing community, not about planning and linking to release notes.
> >
> > Next, I fully agree that it is very good to have isolated IEP describing
> a
> > single major change. However, please note that a lot of IEPs start as
> > research activities. During this time we do not know what correct
> solution
> > is, how many major changes would be needed, how many components will be
> > affected, and how many tickets will be filed. All of this can change
> > drastically during IEP lifetime. Which looks perfectly fine for me - we
> > know what to improve, but do not know yet how exactly. If it is about
> real
> > implementation or planning you can always create umbrella ticket or so.
> > Moreover, it is also OK for IEPs to span multiple releases, e.g. this is
> > how we do this with TDE - huge change, multiple phases, multiple
> releases.
> >
> > About names - again, ideally they should be concrete and non ambiguous.
> But
> > currently most of immediate product goals are around stabilization and
> > performance. This is really where we are. So if I, for example,
> > investigated a set of potential optimizations in area X, why can't I name
> > it "Optimize X"? If someone else will find more optimizations in the same
> > are while the first IEP is still active, he can join this IEP. If he find
> > them after first IEP is completed, then he can name it "Optimize X part
> 2",
> > while "Optimize X" will be in archive directory. So who really cares?
> >
> > I do not see how enforcing any strict rules could help us here.
> >
> > +1 for the rest.
> >
> > On Wed, Nov 7, 2018 at 7:36 PM Maxim Muzafarov <[hidden email]>
> wrote:
> >
> > > Igniters,
> > >
> > > I think, our community have accumulated enough experience with the
> process
> > > of
> > > Ignite Enhancement Proposal (IEP) of introducing the major changes
> > > into the Apache
> > > Ignite. Now we have to take one step forward and make every major
> change
> > > and\or
> > > improvement clear not only for community developers but also for the
> > > Apache Ignite
> > > users too.
> > >
> > > Currently, I've seen two different strategies with creating IEPs:
> > >  - Single IEP per single major improvement;
> > >  - Single IEP per unit of functionality (group a few major
> improvements);
> > >
> > > Both of them have different advantages and disadvantages.
> > >
> > >
> > > Let's remove the ambiguity and build a convenient working process. For
> > > example,
> > > we may consider the best practices used in other open-source projects
> [2]
> > > [3].
> > >
> > > I propose to (and also propose to update the corresponding wiki page
> [1]
> > > and our
> > > community processes):
> > >  - use to "Single IEP per single major improvement" approach;
> > >  - do not group the major enhancements into the single IEP;
> > >  - avoid inaccurate proposal names (e.g. optimization, improvement,
> > > stabilization of etc.);
> > >  - for any public API changes create the IEP;
> > >  - add a `compatibility\mirgration` section to IEP Template;
> > >  - add a `public API changes` section to the IEP Template;
> > >  - link each major release note with the corresponding IEP page (users
> > > will have a
> > >    better understanding of each feature).
> > >
> > >  Igniters, please, share your thoughts.
> > >
> > >
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Enhancement+Proposal
> > > [2]
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > [3]
> > >
> https://cwiki.apache.org/confluence/display/SAMZA/Samza+Enhancement+Proposal
> > >
>

--
Best regards,
Ivan Pavlukhin