Ignite Release Schedules - a suggestion

[endianignite]

Dear Igniters,

First of all, congratulations on heading towards the 1.6 release.  Each Ignite release is a huge amount of work, done to a very high standard by a great group of people.  I raise my hat to each and every one of you!

I would like to open a discussion regarding release schedules, and how we currently perform releases.  Right now, we have very large waterfall releases, that are a combination of substantial numbers of bug fixes, along with significant new features - binary marshaling from 1.5 is one that I witnessed first hand.  The integration of all of these changes can be challenging, and from my observation tends to extend the time taken to go from the final code freeze to the point of release.  Some data from the release dates:

Release,Date,DaysSinceLastRelease
1.0,02-Apr-15,
1.1,28-May-15,56
1.2,29-Jun-15,32
1.3,21-Jul-15,22
1.4,28-Sep-15,69
1.5,04-Jan-16,98
1.6*,31-May-16,148
1.7+,03-Dec-16,186
1.8+,17-Jul-17,226.7
1.9+,11-Apr-18,267.4
2.0+,13-Feb-19,308.1

*estimated date
+extrapolated date.  Equation: date=407*releaseVersion-505.9 taken from Excel best-fit line, R^2=0.9906.  

As you can see, the time between releases is trending upwards since 1.3, and on a very steep gradient.  Doing some (admittedly silly) extrapolation, at the current it could take 308 days to go from a future version 1.9 to a 2.0 release, just assuming the current growth rate.  This could mean that a bug found in April 2018 would not be released until February 2019!  Currently I'm facing a bug (https://issues.apache.org/jira/browse/IGNITE-2080) that was found in Dec-15, fixed in Feb-16, but will not be released until May/June 2015.  

In my opinion this is not sustainable if we want Ignite to grow in userbase and functionality, whilst retaining a reputation for stability and being bug-free, as I'm sure we all do.  Therefore, as a sign of Ignite's growing maturity, I would like to suggest that we adopt a release schedule where bug fixes are released on a timed-release basis, and new features are included in less frequent major releases.  There are many examples of this kind of release schedule: Intel's tick-tock cycle, Ubuntu's timed-release cycle, Redhat's major/minor/asynch cycle.  

I propose that the release schedule for Ignite would be separated in to two streams: bug fixes and major features.  The first stream, bug fixes, would close for code freeze every month, undergo testing and QA and then be released, hopefully within two weeks of the code freeze.  Bugs not fixed in time for the code freeze would be pushed to the next monthly cycle.  The Major Features stream would be over a significantly longer period, although I would still suggest that it be less than the current 3-5 month interval.  

Finally, if you got this far then well done, you have got staying power.  I hope that you will take the above thoughts and analysis in the way that they are offered, as a positive, and not as criticism (which they are not).

I'm really interested to hear your thoughts on this subject.  I'm happy to try and lead a release cycle change if the community would like me to, or to assist someone within GridGain if that makes more sense.

Kind regards
Mike

[dsetrakyan]

How will it look branch-wise? Currently, if a feature passes a review, it
ends up in master, together with bug fixes. If we were to separate bug
fixes from features, then we need a different branching structure. For
example, we may have another release branch which will get cherry-picked
bug fixes, but not the features.

Is this what you had in mind?

D.

On Fri, May 13, 2016 at 2:21 AM, endianignite <[hidden email]>
wrote:

> Dear Igniters,
>
> First of all, congratulations on heading towards the 1.6 release.  Each
> Ignite release is a huge amount of work, done to a very high standard by a
> great group of people.  I raise my hat to each and every one of you!
>
> I would like to open a discussion regarding release schedules, and how we
> currently perform releases.  Right now, we have very large waterfall
> releases, that are a combination of substantial numbers of bug fixes, along
> with significant new features - binary marshaling from 1.5 is one that I
> witnessed first hand.  The integration of all of these changes can be
> challenging, and from my observation tends to extend the time taken to go
> from the final code freeze to the point of release.  Some data from the
> release dates:
>
> Release,Date,DaysSinceLastRelease
> 1.0,02-Apr-15,
> 1.1,28-May-15,56
> 1.2,29-Jun-15,32
> 1.3,21-Jul-15,22
> 1.4,28-Sep-15,69
> 1.5,04-Jan-16,98
> 1.6*,31-May-16,148
> 1.7+,03-Dec-16,186
> 1.8+,17-Jul-17,226.7
> 1.9+,11-Apr-18,267.4
> 2.0+,13-Feb-19,308.1
>
> *estimated date
> +extrapolated date.  Equation: date=407*releaseVersion-505.9 taken from
> Excel best-fit line, R^2=0.9906.
>
> As you can see, the time between releases is trending upwards since 1.3,
> and
> on a very steep gradient.  Doing some (admittedly silly) extrapolation, at
> the current it could take 308 days to go from a future version 1.9 to a 2.0
> release, just assuming the current growth rate.  This could mean that a bug
> found in April 2018 would not be released until February 2019!  Currently
> I'm facing a bug (https://issues.apache.org/jira/browse/IGNITE-2080) that
> was found in Dec-15, fixed in Feb-16, but will not be released until
> May/June 2015.
>
> In my opinion this is not sustainable if we want Ignite to grow in userbase
> and functionality, whilst retaining a reputation for stability and being
> bug-free, as I'm sure we all do.  Therefore, as a sign of Ignite's growing
> maturity, I would like to suggest that we adopt a release schedule where
> bug
> fixes are released on a timed-release basis, and new features are included
> in less frequent major releases.  There are many examples of this kind of
> release schedule: Intel's tick-tock cycle, Ubuntu's timed-release cycle,
> Redhat's major/minor/asynch cycle.
>
> I propose that the release schedule for Ignite would be separated in to two
> streams: bug fixes and major features.  The first stream, bug fixes, would
> close for code freeze every month, undergo testing and QA and then be
> released, hopefully within two weeks of the code freeze.  Bugs not fixed in
> time for the code freeze would be pushed to the next monthly cycle.  The
> Major Features stream would be over a significantly longer period, although
> I would still suggest that it be less than the current 3-5 month interval.
>
> Finally, if you got this far then well done, you have got staying power.  I
> hope that you will take the above thoughts and analysis in the way that
> they
> are offered, as a positive, and not as criticism (which they are not).
>
> I'm really interested to hear your thoughts on this subject.  I'm happy to
> try and lead a release cycle change if the community would like me to, or
> to
> assist someone within GridGain if that makes more sense.
>
> Kind regards
> Mike
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Release-Schedules-a-suggestion-tp8867.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

[Konstantin Boudnik-2]

I don't think the extrapolation works in this particular case, although it
looks scary ;) However, I agree, shorter bug fix & smaller feature releases
might make a lot of sense, considering how actively the userbase and community
grow. However, major features take longer time to mature and be integrated
into the master, hence they should be carried on the branches (an good example
such is IGNITE-1371). And this might lead, in the extreme case, to the
multi-release universe.

Hence coming Dmitriy's quesion about branching model. So I would like to, once
again, offer
    http://nvie.com/posts/a-successful-git-branching-model/

which effectively and transparently allow to support multiple release trains.

But in general, releases shouldn't be a big deal and any committer can call a
release at any given moment. All it needs to pass is >3 PMC votes to become an
official one. Easier the release process is and more release managers (RMs) we
have - shorter the release spam might become.

Cos

On Fri, May 13, 2016 at 02:21AM, endianignite wrote:

> Dear Igniters,
>
> First of all, congratulations on heading towards the 1.6 release.  Each
> Ignite release is a huge amount of work, done to a very high standard by a
> great group of people.  I raise my hat to each and every one of you!
>
> I would like to open a discussion regarding release schedules, and how we
> currently perform releases.  Right now, we have very large waterfall
> releases, that are a combination of substantial numbers of bug fixes, along
> with significant new features - binary marshaling from 1.5 is one that I
> witnessed first hand.  The integration of all of these changes can be
> challenging, and from my observation tends to extend the time taken to go
> from the final code freeze to the point of release.  Some data from the
> release dates:
>
> Release,Date,DaysSinceLastRelease
> 1.0,02-Apr-15,
> 1.1,28-May-15,56
> 1.2,29-Jun-15,32
> 1.3,21-Jul-15,22
> 1.4,28-Sep-15,69
> 1.5,04-Jan-16,98
> 1.6*,31-May-16,148
> 1.7+,03-Dec-16,186
> 1.8+,17-Jul-17,226.7
> 1.9+,11-Apr-18,267.4
> 2.0+,13-Feb-19,308.1
>
> *estimated date
> +extrapolated date.  Equation: date=407*releaseVersion-505.9 taken from
> Excel best-fit line, R^2=0.9906.  
>
> As you can see, the time between releases is trending upwards since 1.3, and
> on a very steep gradient.  Doing some (admittedly silly) extrapolation, at
> the current it could take 308 days to go from a future version 1.9 to a 2.0
> release, just assuming the current growth rate.  This could mean that a bug
> found in April 2018 would not be released until February 2019!  Currently
> I'm facing a bug (https://issues.apache.org/jira/browse/IGNITE-2080) that
> was found in Dec-15, fixed in Feb-16, but will not be released until
> May/June 2015.  
>
> In my opinion this is not sustainable if we want Ignite to grow in userbase
> and functionality, whilst retaining a reputation for stability and being
> bug-free, as I'm sure we all do.  Therefore, as a sign of Ignite's growing
> maturity, I would like to suggest that we adopt a release schedule where bug
> fixes are released on a timed-release basis, and new features are included
> in less frequent major releases.  There are many examples of this kind of
> release schedule: Intel's tick-tock cycle, Ubuntu's timed-release cycle,
> Redhat's major/minor/asynch cycle.  
>
> I propose that the release schedule for Ignite would be separated in to two
> streams: bug fixes and major features.  The first stream, bug fixes, would
> close for code freeze every month, undergo testing and QA and then be
> released, hopefully within two weeks of the code freeze.  Bugs not fixed in
> time for the code freeze would be pushed to the next monthly cycle.  The
> Major Features stream would be over a significantly longer period, although
> I would still suggest that it be less than the current 3-5 month interval.  
>
> Finally, if you got this far then well done, you have got staying power.  I
> hope that you will take the above thoughts and analysis in the way that they
> are offered, as a positive, and not as criticism (which they are not).
>
> I'm really interested to hear your thoughts on this subject.  I'm happy to
> try and lead a release cycle change if the community would like me to, or to
> assist someone within GridGain if that makes more sense.
>
> Kind regards
> Mike
>
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Release-Schedules-a-suggestion-tp8867.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[endianignite]

With regard to branch structure, I took some time to think and have a suggestion.  Feel free to pick holes in it.

Major feature releases would continue to have major version numbers: e.g., 1.7, 1.8, 2.x and so on.  Bug-fix releases would be minor releases: e.g., 1.7.1, 1.7.2, 1.7.x, etc.  Major releases get their own branch, although it will probably be sensible to keep major features on their own JIRA branch before being merged on to the release branch.  Once a major release "goes gold", bug fixes would be continued on that release branch, and the next major release would go to a new branch.  Bug fixes on the current "gold" branch would be merged regularly in to the new major release branch, until that new major release branch is ready to "go gold", at which point the cycle would repeat, and new bug fixes would go on to the newly released major release branch.

In the above scheme, master becomes deprecated, but there is still only one major release branch that is "gold" at any one time.  We would not be in a situation where 1.7.x was still being developed on once 1.8.x has been released.

[Konstantin Boudnik-2]

On Tue, May 17, 2016 at 01:09AM, endianignite wrote:

> With regard to branch structure, I took some time to think and have a
> suggestion.  Feel free to pick holes in it.
>
> Major feature releases would continue to have major version numbers: e.g.,
> 1.7, 1.8, 2.x and so on.  Bug-fix releases would be minor releases: e.g.,
> 1.7.1, 1.7.2, 1.7.x, etc.  Major releases get their own branch, although it
> will probably be sensible to keep major features on their own JIRA branch
> before being merged on to the release branch.  Once a major release "goes
> gold", bug fixes would be continued on that release branch, and the next
> major release would go to a new branch.  Bug fixes on the current "gold"
> branch would be merged regularly in to the new major release branch, until
> that new major release branch is ready to "go gold", at which point the
> cycle would repeat, and new bug fixes would go on to the newly released
> major release branch.

How would you propagate bug fixed among releases in this case?

> In the above scheme, master becomes deprecated, but there is still only one
> major release branch that is "gold" at any one time.  We would not be in a
> situation where 1.7.x was still being developed on once 1.8.x has been
> released.
>
> <http://apache-ignite-developers.2346864.n4.nabble.com/file/n8939/Ignite_Source_Control.jpg>
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Release-Schedules-a-suggestion-tp8867p8939.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[dsetrakyan]

I actually think that the structure can be simpler and we should just
create a branch per release. Most likely, everything that is currently in
master should be released anyway, as we all work very hard on making sure
that master branch is always stable.

I think it does make sense to have a release at least once a quarter or
even more often. It looks like we are all in agreement here, no?

D.


On Tue, May 17, 2016 at 5:21 PM, Konstantin Boudnik <[hidden email]> wrote:

> On Tue, May 17, 2016 at 01:09AM, endianignite wrote:
> > With regard to branch structure, I took some time to think and have a
> > suggestion.  Feel free to pick holes in it.
> >
> > Major feature releases would continue to have major version numbers:
> e.g.,
> > 1.7, 1.8, 2.x and so on.  Bug-fix releases would be minor releases: e.g.,
> > 1.7.1, 1.7.2, 1.7.x, etc.  Major releases get their own branch, although
> it
> > will probably be sensible to keep major features on their own JIRA branch
> > before being merged on to the release branch.  Once a major release "goes
> > gold", bug fixes would be continued on that release branch, and the next
> > major release would go to a new branch.  Bug fixes on the current "gold"
> > branch would be merged regularly in to the new major release branch,
> until
> > that new major release branch is ready to "go gold", at which point the
> > cycle would repeat, and new bug fixes would go on to the newly released
> > major release branch.
>
> How would you propagate bug fixed among releases in this case?
>
> > In the above scheme, master becomes deprecated, but there is still only
> one
> > major release branch that is "gold" at any one time.  We would not be in
> a
> > situation where 1.7.x was still being developed on once 1.8.x has been
> > released.
> >
> > <
> http://apache-ignite-developers.2346864.n4.nabble.com/file/n8939/Ignite_Source_Control.jpg
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Release-Schedules-a-suggestion-tp8867p8939.html
> > Sent from the Apache Ignite Developers mailing list archive at
> Nabble.com.
>

[endianignite]

Konstantin Boudnik-2 wrote

On Tue, May 17, 2016 at 01:09AM, endianignite wrote:
[...]
How would you propagate bug fixed among releases in this case?
signature.asc (237 bytes) <http://apache-ignite-developers.2346864.n4.nabble.com/attachment/8959/0/signature.asc>

I think as shown on the diagram, bug fixes that are made in 1.7.x releases would be merged into the release branch that is being prepared for 1.8.  Once the new 1.8 release branch goes "gold", any further bug fixes go in to 1.8 (and 1.7.x releases are deprecated).

[endianignite]

dsetrakyan wrote

I actually think that the structure can be simpler and we should just
create a branch per release. Most likely, everything that is currently in
master should be released anyway, as we all work very hard on making sure
that master branch is always stable.

I think it does make sense to have a release at least once a quarter or
even more often. It looks like we are all in agreement here, no?

D.

My feeling is that a big release every quarter is still not allowing us to be agile enough.  If there is a breaking issue that is discovered, but we have merged half of an incomplete big feature in to master, then we're going to have a serious problem.  My own issue at the moment, IGNITE-2080, is a good example.  Ignite 1.5 cannot run on SunOS, and this turns out to be a serious issue for my project, as my first client's application will only run on SunOS!  We are having to provision extra hardware and figure out whether my project can be run on a different, untested environment, all of which is impacting the release schedules of my project, my first client's project, and the projects of those who are dependent on that first client.  The knock-on effect can be very damaging.

I'm very keen to keep things as simple as possible - but no simpler :-)