Introducing "compatibility level" concept.

[Vladimir Ozerov]

Igniters,

Normally we are trying to maintain backward compatibility with previous
versions. But it is not always possible.

E.g. we are about to release portable protocol. There are lots suggestions
how to optimize it, but all of them are relatively hard to implement. It
would not be a problem if are able to improve it iteratively from release
to release while still allowing for different versions (e.g. 1.5 and 1.6)
to communicate.

What if we add a top-level property "*compatibility level*" allowing user
to "downgrade" some parts of the system to communicate with earlier
versions?

Vladimir.

[Branko Čibej]

On 30.09.2015 09:51, Vladimir Ozerov wrote:

> Igniters,
>
> Normally we are trying to maintain backward compatibility with previous
> versions. But it is not always possible.
>
> E.g. we are about to release portable protocol. There are lots suggestions
> how to optimize it, but all of them are relatively hard to implement. It
> would not be a problem if are able to improve it iteratively from release
> to release while still allowing for different versions (e.g. 1.5 and 1.6)
> to communicate.
>
> What if we add a top-level property "*compatibility level*" allowing user
> to "downgrade" some parts of the system to communicate with earlier
> versions?

That's likely to be a huge pain to maintain. The problem is that it
doesn't force you to think about compatibility too much, and you'll end
up having any number of incompatible protocols that you'll have to
maintain simultaneously.

There's an IMO better way: define the protocol to be natrually
extensible, and introduce a capability exchange step. Map protocol
extensions to capabilities (these can represented as simple tokens, or
whatever), then make the client and server use any capabilities that are
common to both.

We do this in Subversion; that's how a 1.9 server can work with a 1.0
client but the same server will work much better (with more features
available) with a 1.9 client. And the other way around, of course.

-- Brane

[Vladimir Ozerov]

Brane,

I see you point, but I do not see how we can implement it in our
distributed environment. Very weird situations could appear. E.g. if there
are two versions A and B:
1) Node A starts.
2) Node B starts and agrees with A to use old A-protocol.
3) Some data is persisted on disk using A-protocol;
4) Cluster shuts down.
5) Several new B-nodes appear, but have no clue that something was
persisted using A-protocol. They decide to use B-protocol.
6) Node A restarts and meets unknown B-protocol.

On Wed, Sep 30, 2015 at 11:01 AM, Branko Čibej <[hidden email]> wrote:

> On 30.09.2015 09:51, Vladimir Ozerov wrote:
> > Igniters,
> >
> > Normally we are trying to maintain backward compatibility with previous
> > versions. But it is not always possible.
> >
> > E.g. we are about to release portable protocol. There are lots
> suggestions
> > how to optimize it, but all of them are relatively hard to implement. It
> > would not be a problem if are able to improve it iteratively from release
> > to release while still allowing for different versions (e.g. 1.5 and 1.6)
> > to communicate.
> >
> > What if we add a top-level property "*compatibility level*" allowing user
> > to "downgrade" some parts of the system to communicate with earlier
> > versions?
>
> That's likely to be a huge pain to maintain. The problem is that it
> doesn't force you to think about compatibility too much, and you'll end
> up having any number of incompatible protocols that you'll have to
> maintain simultaneously.
>
> There's an IMO better way: define the protocol to be natrually
> extensible, and introduce a capability exchange step. Map protocol
> extensions to capabilities (these can represented as simple tokens, or
> whatever), then make the client and server use any capabilities that are
> common to both.
>
> We do this in Subversion; that's how a 1.9 server can work with a 1.0
> client but the same server will work much better (with more features
> available) with a 1.9 client. And the other way around, of course.
>
> -- Brane
>

[Branko Čibej]

My point is that is has to be just one extensible protocol. So in your
case, either: in step 6, the node A would be able to read the
A-compatible bits of data; or: in step 5, the B-nodes will know that the
persistent store was made with A.

Doing it any other way will force you to dumb down your whole grid,
caches, etc. to the lowest common denominator; which makes heterogeneous
grids impossible, and therefore makes rolling upgrades on a live grid
impossible.

Of course you may not care about these scenarios.

-- Brane

On 30.09.2015 10:12, Vladimir Ozerov wrote:

> Brane,
>
> I see you point, but I do not see how we can implement it in our
> distributed environment. Very weird situations could appear. E.g. if there
> are two versions A and B:
> 1) Node A starts.
> 2) Node B starts and agrees with A to use old A-protocol.
> 3) Some data is persisted on disk using A-protocol;
> 4) Cluster shuts down.
> 5) Several new B-nodes appear, but have no clue that something was
> persisted using A-protocol. They decide to use B-protocol.
> 6) Node A restarts and meets unknown B-protocol.
>
> On Wed, Sep 30, 2015 at 11:01 AM, Branko Čibej <[hidden email]> wrote:
>
>> On 30.09.2015 09:51, Vladimir Ozerov wrote:
>>> Igniters,
>>>
>>> Normally we are trying to maintain backward compatibility with previous
>>> versions. But it is not always possible.
>>>
>>> E.g. we are about to release portable protocol. There are lots
>> suggestions
>>> how to optimize it, but all of them are relatively hard to implement. It
>>> would not be a problem if are able to improve it iteratively from release
>>> to release while still allowing for different versions (e.g. 1.5 and 1.6)
>>> to communicate.
>>>
>>> What if we add a top-level property "*compatibility level*" allowing user
>>> to "downgrade" some parts of the system to communicate with earlier
>>> versions?
>> That's likely to be a huge pain to maintain. The problem is that it
>> doesn't force you to think about compatibility too much, and you'll end
>> up having any number of incompatible protocols that you'll have to
>> maintain simultaneously.
>>
>> There's an IMO better way: define the protocol to be natrually
>> extensible, and introduce a capability exchange step. Map protocol
>> extensions to capabilities (these can represented as simple tokens, or
>> whatever), then make the client and server use any capabilities that are
>> common to both.
>>
>> We do this in Subversion; that's how a 1.9 server can work with a 1.0
>> client but the same server will work much better (with more features
>> available) with a 1.9 client. And the other way around, of course.
>>
>> -- Brane
>>

[yzhdanov]

>> 6) Node A restarts and meets unknown B-protocol.

6.1 Node A throws exception on start
6.2 Or node A invalidates persistent store (ignores the prev state)
6.3 Or user should have ability to specify converter somehow. Converter
logic should be provided with release of new protocol (with more recent
versions) so user can plug it in to older versions.

Thoughts?

--Yakov

[Sergi]

For me "compatibility level" looks like a complex and dangerous concept.

It really must be enough to have versioned protocol, when newer protocol
fully understands older ones. And there is no need to store the protocol
version
with each object, because it must be enough to send it on handshake
and store it to persistent storage metadata.

The only issue here is that we are trying to marshal messages only once
during a broadcast while with this approach we must make sure that
such a reuse of bytes happens only when protocol is the same for all nodes
or we must group receiver nodes by protocol version.

Sergi


2015-09-30 12:03 GMT+03:00 Yakov Zhdanov <[hidden email]>:

> >> 6) Node A restarts and meets unknown B-protocol.
>
> 6.1 Node A throws exception on start
> 6.2 Or node A invalidates persistent store (ignores the prev state)
> 6.3 Or user should have ability to specify converter somehow. Converter
> logic should be provided with release of new protocol (with more recent
> versions) so user can plug it in to older versions.
>
> Thoughts?
>
> --Yakov
>

[Raul Kripalani-2]

Exposing a manually adjustable "compatibility level" makes the user a
participant of Ignite's internals – something I dislike. Most users won't
bother with investigating deep enough to make an informed decision to set
this value accordingly, and it'll end up being the typical obscure config
parameter that only a handful of people understand.

I would prefer an intelligent handshake/negotiation like Branko proposed.
Mapping protocol versions to capabilities and intersecting both peers to
find the common set. Some capabilities must be compulsory, others can be
optional.

For those that are optional we may need to define a policy of how Ignite
should act if the user's code attempts to use a capability that's currently
unsupported by the topology -- do we fail with an Exception (blocking
user's code)? Do we define a fallback behaviour for each capability? Do we
ignore the user's attempt to use an unsupported capability?

While this level of dynamic negotiation sounds appealing, it can lead to
extremely unpredictable results. Imagine we have nodes A, B, C, all on
v3.0. A client node dispatches a ComputeJob that uses some new
functionality introduced in v3.0: it executes correctly. Now node D joins
and dumbs down the whole grid to 2.0. The next time the client dispatches
the same job, it'll fail.

Needless to say that the effort to maintain infinite compatibility policies
between versions is going to be huge and it'll lead to technical baggage.

So some things to consider:

* Don't make the user a participant of the internal protocol. Instead,
let's make the community agree on a backwards compatibility policy, e.g.
let's declare that major versions of Ignite will NOT be compatible with old
major versions (1.x vs 2.x), and that minor versions will only be
compatible two steps back, e.g. (1.4.x will be compatible at most with
1.2.x, but not with 1.1.x). This will also gently push users to keep
upgrading and not dwelling in old versions because they're compatible with
everything.

* Separate the different interoperability protocols we have in Ignite:
communication, persistence, etc. And make those versionable independently
of one another.

* How about a "Grid stabilisation period" parameter? A window of grace
(e.g. 30 min) during which existing nodes will dumb down to the common
denominator. After that window elapses, no further dumbing down will be
allowed. New nodes attempting to join and trigger further dumbing down will
be rejected.

At the end of the day, I doubt that users running Ignite 2.1.0 will
suddenly want to join an Ignite 1.0.0 node 3 days after the grid has
started...

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Wed, Sep 30, 2015 at 9:28 AM, Branko Čibej <[hidden email]> wrote:

> My point is that is has to be just one extensible protocol. So in your
> case, either: in step 6, the node A would be able to read the
> A-compatible bits of data; or: in step 5, the B-nodes will know that the
> persistent store was made with A.
>
> Doing it any other way will force you to dumb down your whole grid,
> caches, etc. to the lowest common denominator; which makes heterogeneous
> grids impossible, and therefore makes rolling upgrades on a live grid
> impossible.
>
> Of course you may not care about these scenarios.
>
> -- Brane
>
> On 30.09.2015 10:12, Vladimir Ozerov wrote:
> > Brane,
> >
> > I see you point, but I do not see how we can implement it in our
> > distributed environment. Very weird situations could appear. E.g. if
> there
> > are two versions A and B:
> > 1) Node A starts.
> > 2) Node B starts and agrees with A to use old A-protocol.
> > 3) Some data is persisted on disk using A-protocol;
> > 4) Cluster shuts down.
> > 5) Several new B-nodes appear, but have no clue that something was
> > persisted using A-protocol. They decide to use B-protocol.
> > 6) Node A restarts and meets unknown B-protocol.
> >
> > On Wed, Sep 30, 2015 at 11:01 AM, Branko Čibej <[hidden email]> wrote:
> >
> >> On 30.09.2015 09:51, Vladimir Ozerov wrote:
> >>> Igniters,
> >>>
> >>> Normally we are trying to maintain backward compatibility with previous
> >>> versions. But it is not always possible.
> >>>
> >>> E.g. we are about to release portable protocol. There are lots
> >> suggestions
> >>> how to optimize it, but all of them are relatively hard to implement.
> It
> >>> would not be a problem if are able to improve it iteratively from
> release
> >>> to release while still allowing for different versions (e.g. 1.5 and
> 1.6)
> >>> to communicate.
> >>>
> >>> What if we add a top-level property "*compatibility level*" allowing
> user
> >>> to "downgrade" some parts of the system to communicate with earlier
> >>> versions?
> >> That's likely to be a huge pain to maintain. The problem is that it
> >> doesn't force you to think about compatibility too much, and you'll end
> >> up having any number of incompatible protocols that you'll have to
> >> maintain simultaneously.
> >>
> >> There's an IMO better way: define the protocol to be natrually
> >> extensible, and introduce a capability exchange step. Map protocol
> >> extensions to capabilities (these can represented as simple tokens, or
> >> whatever), then make the client and server use any capabilities that are
> >> common to both.
> >>
> >> We do this in Subversion; that's how a 1.9 server can work with a 1.0
> >> client but the same server will work much better (with more features
> >> available) with a 1.9 client. And the other way around, of course.
> >>
> >> -- Brane
> >>
>
>

[Konstantin Boudnik-2]

+1
Any sort of overly nice infra-software ends up being either
 - an disaster integration and operation disaster
 - a maintainer nightmares because of complex internal logic
 - a dangerously silent data-corruption contraption
or any combination of the above.

Hadoop RPC versioning is somewhat close to the compatibility level concept here. And ot even simpler, I believe. Yet, it makes downstream development to be complex, bulky, hard to test, and error prone _even_ with right tools like Bigtop.

Perhaps, it would make sense to reconsider the use case that lead to the proposal and think of different ways to solve it.

Cos

On September 30, 2015 1:28:52 AM PDT, "Branko Čibej" <[hidden email]> wrote:

>My point is that is has to be just one extensible protocol. So in your
>case, either: in step 6, the node A would be able to read the
>A-compatible bits of data; or: in step 5, the B-nodes will know that
>the
>persistent store was made with A.
>
>Doing it any other way will force you to dumb down your whole grid,
>caches, etc. to the lowest common denominator; which makes
>heterogeneous
>grids impossible, and therefore makes rolling upgrades on a live grid
>impossible.
>
>Of course you may not care about these scenarios.
>
>-- Brane
>
>On 30.09.2015 10:12, Vladimir Ozerov wrote:
>> Brane,
>>
>> I see you point, but I do not see how we can implement it in our
>> distributed environment. Very weird situations could appear. E.g. if
>there
>> are two versions A and B:
>> 1) Node A starts.
>> 2) Node B starts and agrees with A to use old A-protocol.
>> 3) Some data is persisted on disk using A-protocol;
>> 4) Cluster shuts down.
>> 5) Several new B-nodes appear, but have no clue that something was
>> persisted using A-protocol. They decide to use B-protocol.
>> 6) Node A restarts and meets unknown B-protocol.
>>
>> On Wed, Sep 30, 2015 at 11:01 AM, Branko Čibej <[hidden email]>
>wrote:
>>
>>> On 30.09.2015 09:51, Vladimir Ozerov wrote:
>>>> Igniters,
>>>>
>>>> Normally we are trying to maintain backward compatibility with
>previous
>>>> versions. But it is not always possible.
>>>>
>>>> E.g. we are about to release portable protocol. There are lots
>>> suggestions
>>>> how to optimize it, but all of them are relatively hard to
>implement. It
>>>> would not be a problem if are able to improve it iteratively from
>release
>>>> to release while still allowing for different versions (e.g. 1.5
>and 1.6)
>>>> to communicate.
>>>>
>>>> What if we add a top-level property "*compatibility level*"
>allowing user
>>>> to "downgrade" some parts of the system to communicate with earlier
>>>> versions?
>>> That's likely to be a huge pain to maintain. The problem is that it
>>> doesn't force you to think about compatibility too much, and you'll
>end
>>> up having any number of incompatible protocols that you'll have to
>>> maintain simultaneously.
>>>
>>> There's an IMO better way: define the protocol to be natrually
>>> extensible, and introduce a capability exchange step. Map protocol
>>> extensions to capabilities (these can represented as simple tokens,
>or
>>> whatever), then make the client and server use any capabilities that
>are
>>> common to both.
>>>
>>> We do this in Subversion; that's how a 1.9 server can work with a
>1.0
>>> client but the same server will work much better (with more features
>>> available) with a 1.9 client. And the other way around, of course.
>>>
>>> -- Brane
>>>

[Vladimir Ozerov]

Guys,

Thank you for feedback. I agree that we should avoid any explicit versions
in configuration if possible. But the question on how to achieve portable
protocol versioning remains open.

I think for initial portables release there is no need to implement
versioning support in the whole Ignite product. It should be enough to
leave a room for changes on protocol level. E.g., by writting a version in
the header of every portable object. 1-2 bytes should be enough.

Thouhgts?

Vladimir.

On Wed, Sep 30, 2015 at 4:40 PM, Konstantin Boudnik <[hidden email]> wrote:

> +1
> Any sort of overly nice infra-software ends up being either
>  - an disaster integration and operation disaster
>  - a maintainer nightmares because of complex internal logic
>  - a dangerously silent data-corruption contraption
> or any combination of the above.
>
> Hadoop RPC versioning is somewhat close to the compatibility level concept
> here. And ot even simpler, I believe. Yet, it makes downstream development
> to be complex, bulky, hard to test, and error prone _even_ with right tools
> like Bigtop.
>
> Perhaps, it would make sense to reconsider the use case that lead to the
> proposal and think of different ways to solve it.
>
> Cos
>
> On September 30, 2015 1:28:52 AM PDT, "Branko Čibej" <[hidden email]>
> wrote:
> >My point is that is has to be just one extensible protocol. So in your
> >case, either: in step 6, the node A would be able to read the
> >A-compatible bits of data; or: in step 5, the B-nodes will know that
> >the
> >persistent store was made with A.
> >
> >Doing it any other way will force you to dumb down your whole grid,
> >caches, etc. to the lowest common denominator; which makes
> >heterogeneous
> >grids impossible, and therefore makes rolling upgrades on a live grid
> >impossible.
> >
> >Of course you may not care about these scenarios.
> >
> >-- Brane
> >
> >On 30.09.2015 10:12, Vladimir Ozerov wrote:
> >> Brane,
> >>
> >> I see you point, but I do not see how we can implement it in our
> >> distributed environment. Very weird situations could appear. E.g. if
> >there
> >> are two versions A and B:
> >> 1) Node A starts.
> >> 2) Node B starts and agrees with A to use old A-protocol.
> >> 3) Some data is persisted on disk using A-protocol;
> >> 4) Cluster shuts down.
> >> 5) Several new B-nodes appear, but have no clue that something was
> >> persisted using A-protocol. They decide to use B-protocol.
> >> 6) Node A restarts and meets unknown B-protocol.
> >>
> >> On Wed, Sep 30, 2015 at 11:01 AM, Branko Čibej <[hidden email]>
> >wrote:
> >>
> >>> On 30.09.2015 09:51, Vladimir Ozerov wrote:
> >>>> Igniters,
> >>>>
> >>>> Normally we are trying to maintain backward compatibility with
> >previous
> >>>> versions. But it is not always possible.
> >>>>
> >>>> E.g. we are about to release portable protocol. There are lots
> >>> suggestions
> >>>> how to optimize it, but all of them are relatively hard to
> >implement. It
> >>>> would not be a problem if are able to improve it iteratively from
> >release
> >>>> to release while still allowing for different versions (e.g. 1.5
> >and 1.6)
> >>>> to communicate.
> >>>>
> >>>> What if we add a top-level property "*compatibility level*"
> >allowing user
> >>>> to "downgrade" some parts of the system to communicate with earlier
> >>>> versions?
> >>> That's likely to be a huge pain to maintain. The problem is that it
> >>> doesn't force you to think about compatibility too much, and you'll
> >end
> >>> up having any number of incompatible protocols that you'll have to
> >>> maintain simultaneously.
> >>>
> >>> There's an IMO better way: define the protocol to be natrually
> >>> extensible, and introduce a capability exchange step. Map protocol
> >>> extensions to capabilities (these can represented as simple tokens,
> >or
> >>> whatever), then make the client and server use any capabilities that
> >are
> >>> common to both.
> >>>
> >>> We do this in Subversion; that's how a 1.9 server can work with a
> >1.0
> >>> client but the same server will work much better (with more features
> >>> available) with a 1.9 client. And the other way around, of course.
> >>>
> >>> -- Brane
> >>>
>