Grid behavior at key deserialization failure during rebalancing

[Anton Vinogradov]

Igniters,

At this moment key deserialization failure during rebalancing cause strange
situation:

Rebalancing from node sent supply message with broken key will be cancelled
at current topology.
All upcoming supply messages from this node will be be ignored, no new
demand messages to this node will be sent.

But when topology will be changed again, node with broken key will take
path at rebalancing again, untill key deserialization failure happen ...
again.

Do we need to improve this situation, and if we have to how should be
handled case with key deserialization failure?

I see some ways:
1) We can inform user about data loss because of deserialization problems,
but keep current rebalancing strategy
2) We can continue rebalancing from this node, but ignore messages with
broken keys. And inform user about data loss.
3) We can pause rebalancing untill deserialization will be fixed somehow,
for example by shutdowning demanding or supplying node.

Thoughts?

[dsetrakyan]

Anton,

I am not sure I fully grok the use case. Can you please explain why a key
can be broken?

D.

On Fri, Feb 12, 2016 at 7:11 AM, Anton Vinogradov <[hidden email]>
wrote:

> Igniters,
>
> At this moment key deserialization failure during rebalancing cause strange
> situation:
>
> Rebalancing from node sent supply message with broken key will be cancelled
> at current topology.
> All upcoming supply messages from this node will be be ignored, no new
> demand messages to this node will be sent.
>
> But when topology will be changed again, node with broken key will take
> path at rebalancing again, untill key deserialization failure happen ...
> again.
>
> Do we need to improve this situation, and if we have to how should be
> handled case with key deserialization failure?
>
> I see some ways:
> 1) We can inform user about data loss because of deserialization problems,
> but keep current rebalancing strategy
> 2) We can continue rebalancing from this node, but ignore messages with
> broken keys. And inform user about data loss.
> 3) We can pause rebalancing untill deserialization will be fixed somehow,
> for example by shutdowning demanding or supplying node.
>
> Thoughts?
>

[Anton Vinogradov]

Dmitriy,

Key can be undeserializable during rebalancing because of many reasons.
For example,
1) It was serialized with errors
2) Deserialization cause error
3) It based on classes unavailable at node trying to deserialize it
Third is the most possible case.


On Sat, Feb 13, 2016 at 3:44 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Anton,
>
> I am not sure I fully grok the use case. Can you please explain why a key
> can be broken?
>
> D.
>
> On Fri, Feb 12, 2016 at 7:11 AM, Anton Vinogradov <
> [hidden email]>
> wrote:
>
> > Igniters,
> >
> > At this moment key deserialization failure during rebalancing cause
> strange
> > situation:
> >
> > Rebalancing from node sent supply message with broken key will be
> cancelled
> > at current topology.
> > All upcoming supply messages from this node will be be ignored, no new
> > demand messages to this node will be sent.
> >
> > But when topology will be changed again, node with broken key will take
> > path at rebalancing again, untill key deserialization failure happen ...
> > again.
> >
> > Do we need to improve this situation, and if we have to how should be
> > handled case with key deserialization failure?
> >
> > I see some ways:
> > 1) We can inform user about data loss because of deserialization
> problems,
> > but keep current rebalancing strategy
> > 2) We can continue rebalancing from this node, but ignore messages with
> > broken keys. And inform user about data loss.
> > 3) We can pause rebalancing untill deserialization will be fixed somehow,
> > for example by shutdowning demanding or supplying node.
> >
> > Thoughts?
> >
>

[dsetrakyan]

Anton,

I am not sure why we would deserialize on the server side with Binary
Marshaller. The data should remain in binary form. Do you know if we have a
test for it?

Thanks,
D.

On Mon, Feb 15, 2016 at 1:20 AM, Anton Vinogradov <[hidden email]>
wrote:

> Dmitriy,
>
> Key can be undeserializable during rebalancing because of many reasons.
> For example,
> 1) It was serialized with errors
> 2) Deserialization cause error
> 3) It based on classes unavailable at node trying to deserialize it
> Third is the most possible case.
>
>
> On Sat, Feb 13, 2016 at 3:44 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Anton,
> >
> > I am not sure I fully grok the use case. Can you please explain why a key
> > can be broken?
> >
> > D.
> >
> > On Fri, Feb 12, 2016 at 7:11 AM, Anton Vinogradov <
> > [hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > At this moment key deserialization failure during rebalancing cause
> > strange
> > > situation:
> > >
> > > Rebalancing from node sent supply message with broken key will be
> > cancelled
> > > at current topology.
> > > All upcoming supply messages from this node will be be ignored, no new
> > > demand messages to this node will be sent.
> > >
> > > But when topology will be changed again, node with broken key will take
> > > path at rebalancing again, untill key deserialization failure happen
> ...
> > > again.
> > >
> > > Do we need to improve this situation, and if we have to how should be
> > > handled case with key deserialization failure?
> > >
> > > I see some ways:
> > > 1) We can inform user about data loss because of deserialization
> > problems,
> > > but keep current rebalancing strategy
> > > 2) We can continue rebalancing from this node, but ignore messages with
> > > broken keys. And inform user about data loss.
> > > 3) We can pause rebalancing untill deserialization will be fixed
> somehow,
> > > for example by shutdowning demanding or supplying node.
> > >
> > > Thoughts?
> > >
> >
>

[Anton Vinogradov]

Dmitry,

I found such behavior at GridCacheRebalancingUnmarshallingFailedSelfTest.
Seems we always unmarshalling keys at supply message handling in case of
OptimizeMarshaller used.
Also it happens when BinaryMarshaller used but key class implements
Externalizable.

On Mon, Feb 15, 2016 at 10:43 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Anton,
>
> I am not sure why we would deserialize on the server side with Binary
> Marshaller. The data should remain in binary form. Do you know if we have a
> test for it?
>
> Thanks,
> D.
>
> On Mon, Feb 15, 2016 at 1:20 AM, Anton Vinogradov <
> [hidden email]>
> wrote:
>
> > Dmitriy,
> >
> > Key can be undeserializable during rebalancing because of many reasons.
> > For example,
> > 1) It was serialized with errors
> > 2) Deserialization cause error
> > 3) It based on classes unavailable at node trying to deserialize it
> > Third is the most possible case.
> >
> >
> > On Sat, Feb 13, 2016 at 3:44 AM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> > > Anton,
> > >
> > > I am not sure I fully grok the use case. Can you please explain why a
> key
> > > can be broken?
> > >
> > > D.
> > >
> > > On Fri, Feb 12, 2016 at 7:11 AM, Anton Vinogradov <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > At this moment key deserialization failure during rebalancing cause
> > > strange
> > > > situation:
> > > >
> > > > Rebalancing from node sent supply message with broken key will be
> > > cancelled
> > > > at current topology.
> > > > All upcoming supply messages from this node will be be ignored, no
> new
> > > > demand messages to this node will be sent.
> > > >
> > > > But when topology will be changed again, node with broken key will
> take
> > > > path at rebalancing again, untill key deserialization failure happen
> > ...
> > > > again.
> > > >
> > > > Do we need to improve this situation, and if we have to how should be
> > > > handled case with key deserialization failure?
> > > >
> > > > I see some ways:
> > > > 1) We can inform user about data loss because of deserialization
> > > problems,
> > > > but keep current rebalancing strategy
> > > > 2) We can continue rebalancing from this node, but ignore messages
> with
> > > > broken keys. And inform user about data loss.
> > > > 3) We can pause rebalancing untill deserialization will be fixed
> > somehow,
> > > > for example by shutdowning demanding or supplying node.
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>

[dsetrakyan]

Anton,

If there is no deserialization for binary marshaller, then I would treat it
as a low priority issue. We should file a ticket and get to it when it
becomes more critical.

D.

On Tue, Feb 16, 2016 at 12:37 AM, Anton Vinogradov <[hidden email]
> wrote:

> Dmitry,
>
> I found such behavior at GridCacheRebalancingUnmarshallingFailedSelfTest.
> Seems we always unmarshalling keys at supply message handling in case of
> OptimizeMarshaller used.
> Also it happens when BinaryMarshaller used but key class implements
> Externalizable.
>
> On Mon, Feb 15, 2016 at 10:43 PM, Dmitriy Setrakyan <[hidden email]
> >
> wrote:
>
> > Anton,
> >
> > I am not sure why we would deserialize on the server side with Binary
> > Marshaller. The data should remain in binary form. Do you know if we
> have a
> > test for it?
> >
> > Thanks,
> > D.
> >
> > On Mon, Feb 15, 2016 at 1:20 AM, Anton Vinogradov <
> > [hidden email]>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > Key can be undeserializable during rebalancing because of many reasons.
> > > For example,
> > > 1) It was serialized with errors
> > > 2) Deserialization cause error
> > > 3) It based on classes unavailable at node trying to deserialize it
> > > Third is the most possible case.
> > >
> > >
> > > On Sat, Feb 13, 2016 at 3:44 AM, Dmitriy Setrakyan <
> > [hidden email]>
> > > wrote:
> > >
> > > > Anton,
> > > >
> > > > I am not sure I fully grok the use case. Can you please explain why a
> > key
> > > > can be broken?
> > > >
> > > > D.
> > > >
> > > > On Fri, Feb 12, 2016 at 7:11 AM, Anton Vinogradov <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > At this moment key deserialization failure during rebalancing cause
> > > > strange
> > > > > situation:
> > > > >
> > > > > Rebalancing from node sent supply message with broken key will be
> > > > cancelled
> > > > > at current topology.
> > > > > All upcoming supply messages from this node will be be ignored, no
> > new
> > > > > demand messages to this node will be sent.
> > > > >
> > > > > But when topology will be changed again, node with broken key will
> > take
> > > > > path at rebalancing again, untill key deserialization failure
> happen
> > > ...
> > > > > again.
> > > > >
> > > > > Do we need to improve this situation, and if we have to how should
> be
> > > > > handled case with key deserialization failure?
> > > > >
> > > > > I see some ways:
> > > > > 1) We can inform user about data loss because of deserialization
> > > > problems,
> > > > > but keep current rebalancing strategy
> > > > > 2) We can continue rebalancing from this node, but ignore messages
> > with
> > > > > broken keys. And inform user about data loss.
> > > > > 3) We can pause rebalancing untill deserialization will be fixed
> > > somehow,
> > > > > for example by shutdowning demanding or supplying node.
> > > > >
> > > > > Thoughts?
> > > > >
> > > >
> > >
> >
>

[Anton Vinogradov]

Done https://issues.apache.org/jira/browse/IGNITE-2660

On Tue, Feb 16, 2016 at 12:58 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Anton,
>
> If there is no deserialization for binary marshaller, then I would treat it
> as a low priority issue. We should file a ticket and get to it when it
> becomes more critical.
>
> D.
>
> On Tue, Feb 16, 2016 at 12:37 AM, Anton Vinogradov <
> [hidden email]
> > wrote:
>
> > Dmitry,
> >
> > I found such behavior at GridCacheRebalancingUnmarshallingFailedSelfTest.
> > Seems we always unmarshalling keys at supply message handling in case of
> > OptimizeMarshaller used.
> > Also it happens when BinaryMarshaller used but key class implements
> > Externalizable.
> >
> > On Mon, Feb 15, 2016 at 10:43 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > wrote:
> >
> > > Anton,
> > >
> > > I am not sure why we would deserialize on the server side with Binary
> > > Marshaller. The data should remain in binary form. Do you know if we
> > have a
> > > test for it?
> > >
> > > Thanks,
> > > D.
> > >
> > > On Mon, Feb 15, 2016 at 1:20 AM, Anton Vinogradov <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > Key can be undeserializable during rebalancing because of many
> reasons.
> > > > For example,
> > > > 1) It was serialized with errors
> > > > 2) Deserialization cause error
> > > > 3) It based on classes unavailable at node trying to deserialize it
> > > > Third is the most possible case.
> > > >
> > > >
> > > > On Sat, Feb 13, 2016 at 3:44 AM, Dmitriy Setrakyan <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Anton,
> > > > >
> > > > > I am not sure I fully grok the use case. Can you please explain
> why a
> > > key
> > > > > can be broken?
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Feb 12, 2016 at 7:11 AM, Anton Vinogradov <
> > > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > At this moment key deserialization failure during rebalancing
> cause
> > > > > strange
> > > > > > situation:
> > > > > >
> > > > > > Rebalancing from node sent supply message with broken key will be
> > > > > cancelled
> > > > > > at current topology.
> > > > > > All upcoming supply messages from this node will be be ignored,
> no
> > > new
> > > > > > demand messages to this node will be sent.
> > > > > >
> > > > > > But when topology will be changed again, node with broken key
> will
> > > take
> > > > > > path at rebalancing again, untill key deserialization failure
> > happen
> > > > ...
> > > > > > again.
> > > > > >
> > > > > > Do we need to improve this situation, and if we have to how
> should
> > be
> > > > > > handled case with key deserialization failure?
> > > > > >
> > > > > > I see some ways:
> > > > > > 1) We can inform user about data loss because of deserialization
> > > > > problems,
> > > > > > but keep current rebalancing strategy
> > > > > > 2) We can continue rebalancing from this node, but ignore
> messages
> > > with
> > > > > > broken keys. And inform user about data loss.
> > > > > > 3) We can pause rebalancing untill deserialization will be fixed
> > > > somehow,
> > > > > > for example by shutdowning demanding or supplying node.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > >
> > > >
> > >
> >
>