Question: different setting for nodes.

[Steshin Vladimir]

Hi, folks.



I recently found that one node might be started with flag
reuse-memory-on-deactivate [1] set off while another node might be
configured with the flag enabled. This ability hinders prediction of
cluster behavior on deactivation/activation. One node keeps data in memory,
other doesn’t.

Should we check this flag is set equally on all nodes? Is this an issue? We
might prevent node start if it has different setting.



[1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE

[Nikolay Izhikov-2]

Vladimir.

This looks like a bug to me.

Can you, please, prepare the simple reproducer for this issue and it’s consequences?

> 5 февр. 2020 г., в 17:08, Vladimir Steshin <[hidden email]> написал(а):
>
> Hi, folks.
>
>
>
> I recently found that one node might be started with flag
> reuse-memory-on-deactivate [1] set off while another node might be
> configured with the flag enabled. This ability hinders prediction of
> cluster behavior on deactivation/activation. One node keeps data in memory,
> other doesn’t.
>
> Should we check this flag is set equally on all nodes? Is this an issue? We
> might prevent node start if it has different setting.
>
>
>
> [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE

[Maxim Muzafarov]

Folks,


I think it's a common problem for such system properties (which may be
different on different nodes and which may lead to unpredictable
cluster behaviour). The same mentioned by Ivan here [1].

Is it better to validate (somehow) all system properties, for
instance, started with IGNITE_ prefix, on all nodes the same way as we
do it for IgniteConfiguration?

[1] https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207

On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov <[hidden email]> wrote:

>
> Vladimir.
>
> This looks like a bug to me.
>
> Can you, please, prepare the simple reproducer for this issue and it’s consequences?
>
>
> > 5 февр. 2020 г., в 17:08, Vladimir Steshin <[hidden email]> написал(а):
> >
> > Hi, folks.
> >
> >
> >
> > I recently found that one node might be started with flag
> > reuse-memory-on-deactivate [1] set off while another node might be
> > configured with the flag enabled. This ability hinders prediction of
> > cluster behavior on deactivation/activation. One node keeps data in memory,
> > other doesn’t.
> >
> > Should we check this flag is set equally on all nodes? Is this an issue? We
> > might prevent node start if it has different setting.
> >
> >
> >
> > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
>

[Alexey Goncharuk]

Vladimir,

In current implementation the flag does not change if data is kept or
cleared from memory - after deactivation the internal structures are
cleared and only data region is kept allocated. If you add a validation for
this flag, you forbid users to do a rolling cluster restart and
enable/disable this behavior without full cluster restart, which itself
means total data loss in case of an in-memory cluster.

Same stands for all other system properties. Unless the property is
absolutely required to be the same on all nodes, there is no need to
validate, this makes the cluster configuration rigid; such properties may
be changed only with the whole cluster downtime.

ср, 5 февр. 2020 г. в 18:40, Maxim Muzafarov <[hidden email]>:

> Folks,
>
>
> I think it's a common problem for such system properties (which may be
> different on different nodes and which may lead to unpredictable
> cluster behaviour). The same mentioned by Ivan here [1].
>
> Is it better to validate (somehow) all system properties, for
> instance, started with IGNITE_ prefix, on all nodes the same way as we
> do it for IgniteConfiguration?
>
> [1]
> https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207
>
> On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Vladimir.
> >
> > This looks like a bug to me.
> >
> > Can you, please, prepare the simple reproducer for this issue and it’s
> consequences?
> >
> >
> > > 5 февр. 2020 г., в 17:08, Vladimir Steshin <[hidden email]>
> написал(а):
> > >
> > > Hi, folks.
> > >
> > >
> > >
> > > I recently found that one node might be started with flag
> > > reuse-memory-on-deactivate [1] set off while another node might be
> > > configured with the flag enabled. This ability hinders prediction of
> > > cluster behavior on deactivation/activation. One node keeps data in
> memory,
> > > other doesn’t.
> > >
> > > Should we check this flag is set equally on all nodes? Is this an
> issue? We
> > > might prevent node start if it has different setting.
> > >
> > >
> > >
> > > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> >
>

[Nikolay Izhikov-2]

Hello, Alexey.

Sorry, my question is not related to the current discussion

> this makes the cluster configuration rigid;

Is it a bad thing to make cluster node configuration rigid?

> 6 февр. 2020 г., в 12:20, Alexey Goncharuk <[hidden email]> написал(а):
>
> Vladimir,
>
> In current implementation the flag does not change if data is kept or
> cleared from memory - after deactivation the internal structures are
> cleared and only data region is kept allocated. If you add a validation for
> this flag, you forbid users to do a rolling cluster restart and
> enable/disable this behavior without full cluster restart, which itself
> means total data loss in case of an in-memory cluster.
>
> Same stands for all other system properties. Unless the property is
> absolutely required to be the same on all nodes, there is no need to
> validate, this makes the cluster configuration rigid; such properties may
> be changed only with the whole cluster downtime.
>
> ср, 5 февр. 2020 г. в 18:40, Maxim Muzafarov <[hidden email]>:
>
>> Folks,
>>
>>
>> I think it's a common problem for such system properties (which may be
>> different on different nodes and which may lead to unpredictable
>> cluster behaviour). The same mentioned by Ivan here [1].
>>
>> Is it better to validate (somehow) all system properties, for
>> instance, started with IGNITE_ prefix, on all nodes the same way as we
>> do it for IgniteConfiguration?
>>
>> [1]
>> https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207
>>
>> On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov <[hidden email]> wrote:
>>>
>>> Vladimir.
>>>
>>> This looks like a bug to me.
>>>
>>> Can you, please, prepare the simple reproducer for this issue and it’s
>> consequences?
>>>
>>>
>>>> 5 февр. 2020 г., в 17:08, Vladimir Steshin <[hidden email]>
>> написал(а):
>>>>
>>>> Hi, folks.
>>>>
>>>>
>>>>
>>>> I recently found that one node might be started with flag
>>>> reuse-memory-on-deactivate [1] set off while another node might be
>>>> configured with the flag enabled. This ability hinders prediction of
>>>> cluster behavior on deactivation/activation. One node keeps data in
>> memory,
>>>> other doesn’t.
>>>>
>>>> Should we check this flag is set equally on all nodes? Is this an
>> issue? We
>>>> might prevent node start if it has different setting.
>>>>
>>>>
>>>>
>>>> [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
>>>
>>

[Alexey Goncharuk]

Nikolay,

This sounds like a philosophical question :)

By rigid I mean inability to change some properties without whole cluster
restart, and I do think it's bad. I have a good example in mind - the
number of threads running rebalance. Originally, we added a validation
check for this configuration, so it was impossible to change it without
cluster restart. There was a user who was running a cluster; the volume of
data grew and afterwards they needed to increase rebalance speed. But they
could not do this. Once we removed the validation and added proper handling
of different rebalance threads number, the user managed to restart nodes
one-by-one, wait for rebalance and keep the cluster up and running.

So I think the general direction should be handle automatically as much as
possible and make behavior consistent, rather than introduce additional
restrictions.

[Ilya Kasnacheev]

Hello!

How about we introduce a *developer warning* about this inconsistency, see
what happens in a release with this warning, and then maybe eventually turn
it into error?

Please note that plain error will be breaking existing code which should
make it blocker for minor releases.

Regards,
--
Ilya Kasnacheev


ср, 5 февр. 2020 г. в 18:40, Maxim Muzafarov <[hidden email]>:

> Folks,
>
>
> I think it's a common problem for such system properties (which may be
> different on different nodes and which may lead to unpredictable
> cluster behaviour). The same mentioned by Ivan here [1].
>
> Is it better to validate (somehow) all system properties, for
> instance, started with IGNITE_ prefix, on all nodes the same way as we
> do it for IgniteConfiguration?
>
> [1]
> https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207
>
> On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Vladimir.
> >
> > This looks like a bug to me.
> >
> > Can you, please, prepare the simple reproducer for this issue and it’s
> consequences?
> >
> >
> > > 5 февр. 2020 г., в 17:08, Vladimir Steshin <[hidden email]>
> написал(а):
> > >
> > > Hi, folks.
> > >
> > >
> > >
> > > I recently found that one node might be started with flag
> > > reuse-memory-on-deactivate [1] set off while another node might be
> > > configured with the flag enabled. This ability hinders prediction of
> > > cluster behavior on deactivation/activation. One node keeps data in
> memory,
> > > other doesn’t.
> > >
> > > Should we check this flag is set equally on all nodes? Is this an
> issue? We
> > > might prevent node start if it has different setting.
> > >
> > >
> > >
> > > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> >
>