Re: Dealing with changing class definitions in Ignite

[Ilya Kasnacheev]

Hello!

I have found that you can avoid compute job type incompatibility problem
(please see history below) by setting

                .setMarshaller(new OptimizedMarshaller())

in your Ignite configuration on all nodes.

However, it is not clear at all why this is needed. Anybody can help? Why
compute jobs are processed by capricious BinaryMarshaller?

Regards,
--
Ilya Kasnacheev


ср, 19 дек. 2018 г. в 02:29, Gert Dubois <[hidden email]>:

> Hey Ilya,
>
> I opened a ticket on the ignite Jira about it.
> https://issues.apache.org/jira/browse/IGNITE-10717
>
> I attached a zip file containing a maven project with sample code that
> reproduces our issue. Reproducing the issue is rather easy though
>
> 1. Have a client + server, with peer class loading enabled
> 2. Create a simple class that implements IgniteRunnable with some class
> field
> 3. Execute an instance of this class on the ignite cluster
> 4. Change the type of the field
> 5. Recompile and execute again. Now it breaks because the class can't be
> serialized using the binary marshaller.
>
> Code to execute these steps is included in the maven project.
>
> On Tue, Dec 18, 2018 at 12:04 PM Ilya Kasnacheev <
> [hidden email]> wrote:
>
>> Hello!
>>
>> Can you show an example of "Ignite Runnables conflicting on the Binary
>> Marshaller"? As a small code snippet perhaps?
>>
>> Maybe I could recommend something but I lack understanding of your use
>> case.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 17 дек. 2018 г. в 18:12, Gert Dubois <[hidden email]>:
>>
>>> Thanks for the reply.
>>>
>>> Everything related to the cache we understand the current architecture,
>>> in our code base we'll probably treat everything cache related the same as
>>> schema migrations in a DB (migration scripts, etc.).
>>> Our real issue is with Ignite Runnables conflicting on the Binary
>>> Marshaller. Every class that gets executed on Ignite as a job gets a stored
>>> definition, this means we can't refactor classes that get used internally
>>> by our Ignite Runnables, because doing so would prevent the updated code
>>> from running. Even worse, if we update our libraries and they changed class
>>> definitions we might run into the same issue, without changing a letter in
>>> our own code. From the documentation it looked like Deployment Mode could
>>> provide a solution for this issue but the Binary Marshaller seems to run
>>> completely separate from the Deployment Mode.
>>>
>>> On Mon, Dec 17, 2018 at 3:18 PM Ilya Kasnacheev <
>>> [hidden email]> wrote:
>>>
>>>> Hello!
>>>>
>>>> As far as my understanding goes:
>>>> You can't peer class load your Key/Value types.
>>>> You also can't redeploy your Key/Value types.
>>>> They even survive node restart via WORKDIR/marshaller directory, and
>>>> come back to haunt you.
>>>>
>>>> There are plans to maybe ease some of those limitations in 3.0, but
>>>> nothing concrete yet. It is not a bug rather a pillar of current Ignite
>>>> architecture. You will have to route around it, such as introducing new
>>>> fields instead of changing types. And maybe avoid having those types on
>>>> server nodes at all, and relying on BinaryObject.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 17 дек. 2018 г. в 15:38, Gert Dubois <[hidden email]>:
>>>>
>>>>> The issue is still present in 2.7.
>>>>> I added a ticket on Jira with sample code that reproduces the issue.
>>>>>
>>>>> https://issues.apache.org/jira/browse/IGNITE-10717
>>>>>
>>>>> For now I think we can work around the issue by overriding the default
>>>>> BinaryNameMapper, but this feels rather hacky to me.
>>>>>
>>>>> On Mon, Dec 17, 2018 at 11:54 AM Denis Mekhanikov <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> Gert,
>>>>>>
>>>>>> Could you check if the problem with a deployment mode reproduces on
>>>>>> Ignite 2.7?
>>>>>> If it does, please file a ticket with an explanation and a reproducer
>>>>>> to https://issues.apache.org/jira/
>>>>>>
>>>>>> Thanks!
>>>>>> Denis
>>>>>>
>>>>>>
>>>>>> пн, 17 дек. 2018 г. в 12:12, Gert Dubois <[hidden email]
>>>>>> >:
>>>>>>
>>>>>>> I investigated the issue further and narrowed the issue down to the
>>>>>>> Binary Marshaller not working as expected given the configured Deployment
>>>>>>> Mode. When forcing my clients to use unique class names in the metadata of
>>>>>>> the Binary Marshaller (I forced this by overriding the global
>>>>>>> BinaryNameMapper and appending a per-client unique suffix to every class
>>>>>>> name) the Deployment Mode behaves as expected. I assume the Binary
>>>>>>> Marshaller keeps a cluster wide state of the Metadata of classes and it
>>>>>>> merges it whenever we serialize a class on a node (regardless of the
>>>>>>> configured Deployment Mode).
>>>>>>> Why is the behaviour of the Binary Marshaller not consistent with
>>>>>>> the way the Deployment Mode works? Is there a cleaner way to solve this,
>>>>>>> besides overriding the BinaryNameMapper?
>>>>>>>
>>>>>>> On Fri, Dec 14, 2018 at 1:07 PM Gert Dubois <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On my current project we are confronted with an issue we are
>>>>>>>> struggling to figure out.
>>>>>>>> We have a simple topology where we have a client node running an
>>>>>>>> IgniteRunnable to a server node. Both nodes have peerClassLoading=true and
>>>>>>>> the default binaryConfiguration with compactFooters=false. The client is
>>>>>>>> started with clientMode=true
>>>>>>>> After execution we shut the client down and refactor the field type
>>>>>>>> of one of the fields in our IgniteRunnable. When we now restart the client
>>>>>>>> we get the following exception:
>>>>>>>>
>>>>>>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Binary
>>>>>>>> type has different field types [typeName=com.trendminer.compute.ClientJob,
>>>>>>>> fieldName=param2, fieldTypeName1=double, fieldTypeName2=int]
>>>>>>>>
>>>>>>>> We tried different Deployment modes, ISOLATED, SHARED/CONTINUOUS
>>>>>>>> with different userVersion. But nothing we change enables us to execute the
>>>>>>>> new class definition on the server node.
>>>>>>>>
>>>>>>>> Ignite Version is 2.4.0
>>>>>>>>
>>>>>>>> Since we are using persistence in our production environment and
>>>>>>>> this causes class definitions for the binary marshaller to be remembered
>>>>>>>> even after cluster restart this is blocking for us since this either forces
>>>>>>>> us to manually clear out the binary data and restart our cluster or it
>>>>>>>> prevents us from refactoring code we we send into Ignite, which is not
>>>>>>>> always possible.
>>>>>>>> So we would like to know what the intended way is to deal with
>>>>>>>> changing class definitions of tasks sent into Ignite?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Gert
>>>>>>>>
>>>>>>>

[dmagda]

Gert, All,

That's not a peer-class-loading issue but rather the binary objects
limitation. It's impossible to change the *type* of an object field. You
can add new fields or remove existing ones. The same limitation applies to
SQL - now way to change the type in runtime. That latter should be
addressed in 2019.

Vladimir, shouldn't it be easier to support fields type change for compute
runnables which are not stored in caches/tables?

--
Denis

On Thu, Dec 20, 2018 at 4:24 AM Gert Dubois <[hidden email]>
wrote:

> For now we can work around the issue by overriding the
> BinaryNameMarshaller and ensuring our clients append a unique identifier
> behind the class name. This ensures that every client gets their own Binary
> Metadata records in the Marshaller for every class they serialize in the
> cluster.
>
> .setNameMapper(new BinaryNameMapper() {
>                     @Override
>                     public String typeName(String clsName) {
>                         if(shouldAddSuffix) {
>                             return clsName + "_clientVersionId";
>                         }
>                         return clsName;
>                     }
>
>                     @Override
>                     public String fieldName(String fieldName) {
>                         return fieldName;
>                     }
>                 });
>
> But we expected the ISOLATED deployment mode would have solved this issue
> for us. It also forces us to maintain some kind of client versioning system
> so when the classes in our jobs change they get a fresh record in the
> Marshaller.
>
> On Wed, Dec 19, 2018 at 6:30 PM Ilya Kasnacheev <[hidden email]>
> wrote:
>
>> Hello!
>>
>> I have found that you can avoid compute job type incompatibility problem
>> (please see history below) by setting
>>
>>                 .setMarshaller(new OptimizedMarshaller())
>>
>> in your Ignite configuration on all nodes.
>>
>> However, it is not clear at all why this is needed. Anybody can help? Why
>> compute jobs are processed by capricious BinaryMarshaller?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 19 дек. 2018 г. в 02:29, Gert Dubois <[hidden email]>:
>>
>>> Hey Ilya,
>>>
>>> I opened a ticket on the ignite Jira about it.
>>> https://issues.apache.org/jira/browse/IGNITE-10717
>>>
>>> I attached a zip file containing a maven project with sample code that
>>> reproduces our issue. Reproducing the issue is rather easy though
>>>
>>> 1. Have a client + server, with peer class loading enabled
>>> 2. Create a simple class that implements IgniteRunnable with some class
>>> field
>>> 3. Execute an instance of this class on the ignite cluster
>>> 4. Change the type of the field
>>> 5. Recompile and execute again. Now it breaks because the class can't be
>>> serialized using the binary marshaller.
>>>
>>> Code to execute these steps is included in the maven project.
>>>
>>> On Tue, Dec 18, 2018 at 12:04 PM Ilya Kasnacheev <
>>> [hidden email]> wrote:
>>>
>>>> Hello!
>>>>
>>>> Can you show an example of "Ignite Runnables conflicting on the Binary
>>>> Marshaller"? As a small code snippet perhaps?
>>>>
>>>> Maybe I could recommend something but I lack understanding of your use
>>>> case.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 17 дек. 2018 г. в 18:12, Gert Dubois <[hidden email]>:
>>>>
>>>>> Thanks for the reply.
>>>>>
>>>>> Everything related to the cache we understand the current
>>>>> architecture, in our code base we'll probably treat everything cache
>>>>> related the same as schema migrations in a DB (migration scripts, etc.).
>>>>> Our real issue is with Ignite Runnables conflicting on the Binary
>>>>> Marshaller. Every class that gets executed on Ignite as a job gets a stored
>>>>> definition, this means we can't refactor classes that get used internally
>>>>> by our Ignite Runnables, because doing so would prevent the updated code
>>>>> from running. Even worse, if we update our libraries and they changed class
>>>>> definitions we might run into the same issue, without changing a letter in
>>>>> our own code. From the documentation it looked like Deployment Mode could
>>>>> provide a solution for this issue but the Binary Marshaller seems to run
>>>>> completely separate from the Deployment Mode.
>>>>>
>>>>> On Mon, Dec 17, 2018 at 3:18 PM Ilya Kasnacheev <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> As far as my understanding goes:
>>>>>> You can't peer class load your Key/Value types.
>>>>>> You also can't redeploy your Key/Value types.
>>>>>> They even survive node restart via WORKDIR/marshaller directory, and
>>>>>> come back to haunt you.
>>>>>>
>>>>>> There are plans to maybe ease some of those limitations in 3.0, but
>>>>>> nothing concrete yet. It is not a bug rather a pillar of current Ignite
>>>>>> architecture. You will have to route around it, such as introducing new
>>>>>> fields instead of changing types. And maybe avoid having those types on
>>>>>> server nodes at all, and relying on BinaryObject.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> пн, 17 дек. 2018 г. в 15:38, Gert Dubois <[hidden email]
>>>>>> >:
>>>>>>
>>>>>>> The issue is still present in 2.7.
>>>>>>> I added a ticket on Jira with sample code that reproduces the issue.
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-10717
>>>>>>>
>>>>>>> For now I think we can work around the issue by overriding the
>>>>>>> default BinaryNameMapper, but this feels rather hacky to me.
>>>>>>>
>>>>>>> On Mon, Dec 17, 2018 at 11:54 AM Denis Mekhanikov <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>> Gert,
>>>>>>>>
>>>>>>>> Could you check if the problem with a deployment mode reproduces on
>>>>>>>> Ignite 2.7?
>>>>>>>> If it does, please file a ticket with an explanation and a
>>>>>>>> reproducer to https://issues.apache.org/jira/
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>
>>>>>>>> пн, 17 дек. 2018 г. в 12:12, Gert Dubois <
>>>>>>>> [hidden email]>:
>>>>>>>>
>>>>>>>>> I investigated the issue further and narrowed the issue down to
>>>>>>>>> the Binary Marshaller not working as expected given the configured
>>>>>>>>> Deployment Mode. When forcing my clients to use unique class names in the
>>>>>>>>> metadata of the Binary Marshaller (I forced this by overriding the global
>>>>>>>>> BinaryNameMapper and appending a per-client unique suffix to every class
>>>>>>>>> name) the Deployment Mode behaves as expected. I assume the Binary
>>>>>>>>> Marshaller keeps a cluster wide state of the Metadata of classes and it
>>>>>>>>> merges it whenever we serialize a class on a node (regardless of the
>>>>>>>>> configured Deployment Mode).
>>>>>>>>> Why is the behaviour of the Binary Marshaller not consistent with
>>>>>>>>> the way the Deployment Mode works? Is there a cleaner way to solve this,
>>>>>>>>> besides overriding the BinaryNameMapper?
>>>>>>>>>
>>>>>>>>> On Fri, Dec 14, 2018 at 1:07 PM Gert Dubois <
>>>>>>>>> [hidden email]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On my current project we are confronted with an issue we are
>>>>>>>>>> struggling to figure out.
>>>>>>>>>> We have a simple topology where we have a client node running an
>>>>>>>>>> IgniteRunnable to a server node. Both nodes have peerClassLoading=true and
>>>>>>>>>> the default binaryConfiguration with compactFooters=false. The client is
>>>>>>>>>> started with clientMode=true
>>>>>>>>>> After execution we shut the client down and refactor the field
>>>>>>>>>> type of one of the fields in our IgniteRunnable. When we now restart the
>>>>>>>>>> client we get the following exception:
>>>>>>>>>>
>>>>>>>>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Binary
>>>>>>>>>> type has different field types [typeName=com.trendminer.compute.ClientJob,
>>>>>>>>>> fieldName=param2, fieldTypeName1=double, fieldTypeName2=int]
>>>>>>>>>>
>>>>>>>>>> We tried different Deployment modes, ISOLATED, SHARED/CONTINUOUS
>>>>>>>>>> with different userVersion. But nothing we change enables us to execute the
>>>>>>>>>> new class definition on the server node.
>>>>>>>>>>
>>>>>>>>>> Ignite Version is 2.4.0
>>>>>>>>>>
>>>>>>>>>> Since we are using persistence in our production environment and
>>>>>>>>>> this causes class definitions for the binary marshaller to be remembered
>>>>>>>>>> even after cluster restart this is blocking for us since this either forces
>>>>>>>>>> us to manually clear out the binary data and restart our cluster or it
>>>>>>>>>> prevents us from refactoring code we we send into Ignite, which is not
>>>>>>>>>> always possible.
>>>>>>>>>> So we would like to know what the intended way is to deal with
>>>>>>>>>> changing class definitions of tasks sent into Ignite?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Gert
>>>>>>>>>>
>>>>>>>>>