Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Akash Shinde
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Ilya Kasnacheev
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:

> Added  [hidden email].
>
> Should I log Jira for this issue?
>
> Thanks,
> Akash
>
>
>
> On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:
>
> > Hi,
> >
> > No both nodes, client and server are running on Ignite 2.7 version. I am
> > starting both server and client from Intellij IDE.
> >
> > Version printed in Server node log:
> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
> >
> > Version in client node log:
> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
> >
> > Thanks,
> > Akash
> >
> > On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
> >
> >> Hi!
> >>
> >> Any chance you might have one node running 2.6 or something like that ?
> >>
> >> It looks like it get a different object that does not match the one
> >> expected in 2.7
> >>
> >> Mikael
> >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
> >>
> >> Before submitting the affinity task ignite first gets the affinity
> cached
> >> function (AffinityInfo) by submitting the cluster wide task
> "AffinityJob".
> >> But while in the process of retrieving the output of this AffinityJob,
> >> ignite deserializes this output. I am getting exception while
> deserailizing
> >> this output.
> >> In TcpDiscoveryNode.readExternal() method while deserailizing the
> >> CacheMetrics object from input stream on 14th iteration I am getting
> >> following exception. Complete stack trace is given in this mail chain.
> >>
> >> Caused by: java.io.IOException: Unexpected error occurred during
> >> unmarshalling of an instance of the class:
> >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
> >>
> >> This is working fine on Ignite 2.6 version but giving problem on 2.7.
> >>
> >> Is this a bug or am I doing something wrong?
> >>
> >> Can someone please help?
> >>
> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> When execute affinity.partition(key), I am getting following exception
> >>> on Ignite  2.7.
> >>>
> >>> Stacktrace:
> >>>
> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
> >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
> >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
> >>> isCancelled=false, retry=null]
> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
> >>> with optimized marshaller
> >>>  at
> >>>
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>  at java.lang.Thread.run(Thread.java:748)
> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
> >>> unmarshal object with optimized marshaller
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
> >>>  at
> >>>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
> >>>  at
> >>>
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
> >>>  ... 10 common frames omitted
> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
> >>> deserialize object with given class loader:
> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
> >>> deserialize object
> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
> >>>  at
> >>>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
> >>>  ... 16 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
> >>>  ... 18 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>>
> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 21 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize field
> >>> [name=assignment]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 27 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>>
> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
> >>>  ... 30 common frames omitted
> >>> Caused by: java.io.IOException: Unexpected error occurred during
> >>> unmarshalling of an instance of the class:
> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
> >>> that all nodes are running the same version of Ignite and that all
> nodes
> >>> have GridOptimizedMarshaller configured with identical optimized
> classes
> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
> your
> >>> serialized classes implement java.io.Externalizable interface, verify
> that
> >>> serialization logic is correct.
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 41 common frames omitted
> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
> >>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
> >>> - Opening socket connection to server
> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
> >>> authenticate using SASL (unknown error)
> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
> >>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
> >>> - Socket connection established to
> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
> >>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7
> has
> >>> expired
> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
> >>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7
> has
> >>> expired, closing socket connection
> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
> >>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
> >>> connectString=qzkpr01.p13.eng.in03.com:50024,
> >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
> >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
> >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
> >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
> >>> 0x3673a59f5dd01c7
> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
> >>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
> >>> - Opening socket connection to server
> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
> >>> authenticate using SASL (unknown error)
> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
> >>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
> >>> - Socket connection established to
> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
> >>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
> >>> - Session establishment complete on server
> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
> >>> 0x3673a59f5dd01ce, negotiated timeout = 10000
> >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
> >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
> >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
> >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
> >>> isCancelled=false, retry=null]
> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
> >>> with optimized marshaller
> >>>  at
> >>>
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>  at java.lang.Thread.run(Thread.java:748)
> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
> >>> unmarshal object with optimized marshaller
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
> >>>  at
> >>>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
> >>>  at
> >>>
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
> >>>  ... 10 common frames omitted
> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
> >>> deserialize object with given class loader:
> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
> >>> deserialize object
> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
> >>>  at
> >>>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
> >>>  ... 16 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
> >>>  ... 18 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>>
> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 21 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize field
> >>> [name=assignment]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 27 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>>
> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
> >>>  ... 30 common frames omitted
> >>> Caused by: java.io.IOException: Unexpected error occurred during
> >>> unmarshalling of an instance of the class:
> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
> >>> that all nodes are running the same version of Ignite and that all
> nodes
> >>> have GridOptimizedMarshaller configured with identical optimized
> classes
> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
> your
> >>> serialized classes implement java.io.Externalizable interface, verify
> that
> >>> serialization logic is correct.
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 41 common frames omitted
> >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
> >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
> >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
> >>> isCancelled=false, retry=null]
> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
> >>> with optimized marshaller
> >>>  at
> >>>
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
> >>>  at
> >>>
> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> >>>  at
> >>>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>  at java.lang.Thread.run(Thread.java:748)
> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
> >>> unmarshal object with optimized marshaller
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
> >>>  at
> >>>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
> >>>  at
> >>>
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
> >>>  ... 10 common frames omitted
> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
> >>> deserialize object with given class loader:
> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
> >>> deserialize object
> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
> >>>  at
> >>>
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> >>>  at
> >>>
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
> >>>  ... 16 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
> >>>  ... 18 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>>
> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 21 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize field
> >>> [name=assignment]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 27 common frames omitted
> >>> Caused by: java.io.IOException: Failed to deserialize object
> >>>
> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
> >>>  ... 30 common frames omitted
> >>> Caused by: java.io.IOException: Unexpected error occurred during
> >>> unmarshalling of an instance of the class:
> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
> >>> that all nodes are running the same version of Ignite and that all
> nodes
> >>> have GridOptimizedMarshaller configured with identical optimized
> classes
> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
> your
> >>> serialized classes implement java.io.Externalizable interface, verify
> that
> >>> serialization logic is correct.
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
> >>>  at
> >>>
> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
> >>>  at
> >>>
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
> >>>  ... 41 common frames omitted
> >>>
> >>> Thanks,
> >>> Akash
> >>>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting error during unmarshalling (Works fine on 2.6)

Prasad Bhalerao
Hi Ilya,

I am not able reproduce this issue in small reproducer project but this is
consistently happening in my project. So I debugged this issue and attached
the screenshot in this mail.



*NOTE:  *This issue occurs if the statistics are enabled on cache
configuration level [cacheCfg.setStatisticsEnabled(true)].



As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All
these cache metrics gets serialized successfully on server node. But at the
time of deserialization on client node only first metrics get de-serialized
successfully but all other metrics till iteration count 13 are
de-serialized as null value and on iteration 14 “ref” byte value in
“OptimizedObjectInputStream.readObject0()” method is read as 81 and code
throws exception.



I think this where it is going wrong. The object copy at the time of
serialization and de-serialization should be same but that’s not happening
in Ignite 2.7 version.

So I debugged this on Ignite 2.6 version. On 2.6 version all 22
cacheMetrics are being de-serialized successfully.


This Looks like a bug to me in serializer/deseriliazer code.



*AffinityJob result being serialized on server:*







*AffinityJob result being de-serialized on client:*




Thanks,
Prasad
On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]>
wrote:

> Hello!
>
> Do you have a reproducer project to reliably confirm this issue?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
>
>> Added  [hidden email].
>>
>> Should I log Jira for this issue?
>>
>> Thanks,
>> Akash
>>
>>
>>
>> On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>> > Hi,
>> >
>> > No both nodes, client and server are running on Ignite 2.7 version. I am
>> > starting both server and client from Intellij IDE.
>> >
>> > Version printed in Server node log:
>> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>> >
>> > Version in client node log:
>> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>> >
>> > Thanks,
>> > Akash
>> >
>> > On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]>
>> wrote:
>> >
>> >> Hi!
>> >>
>> >> Any chance you might have one node running 2.6 or something like that ?
>> >>
>> >> It looks like it get a different object that does not match the one
>> >> expected in 2.7
>> >>
>> >> Mikael
>> >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>> >>
>> >> Before submitting the affinity task ignite first gets the affinity
>> cached
>> >> function (AffinityInfo) by submitting the cluster wide task
>> "AffinityJob".
>> >> But while in the process of retrieving the output of this AffinityJob,
>> >> ignite deserializes this output. I am getting exception while
>> deserailizing
>> >> this output.
>> >> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> >> CacheMetrics object from input stream on 14th iteration I am getting
>> >> following exception. Complete stack trace is given in this mail chain.
>> >>
>> >> Caused by: java.io.IOException: Unexpected error occurred during
>> >> unmarshalling of an instance of the class:
>> >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> >>
>> >> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>> >>
>> >> Is this a bug or am I doing something wrong?
>> >>
>> >> Can someone please help?
>> >>
>> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> When execute affinity.partition(key), I am getting following exception
>> >>> on Ignite  2.7.
>> >>>
>> >>> Stacktrace:
>> >>>
>> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>> >>>  at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>>  at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>>  at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>> >>> unmarshal object with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>> >>>  at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>> >>>  ... 10 common frames omitted
>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>> >>> deserialize object with given class loader:
>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>> >>> deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>> >>>  at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>> >>>  ... 16 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>> >>>  ... 18 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 21 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize field
>> >>> [name=assignment]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 27 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>> >>>  ... 30 common frames omitted
>> >>> Caused by: java.io.IOException: Unexpected error occurred during
>> >>> unmarshalling of an instance of the class:
>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> Check
>> >>> that all nodes are running the same version of Ignite and that all
>> nodes
>> >>> have GridOptimizedMarshaller configured with identical optimized
>> classes
>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
>> your
>> >>> serialized classes implement java.io.Externalizable interface, verify
>> that
>> >>> serialization logic is correct.
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 41 common frames omitted
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Opening socket connection to server
>> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>> >>> authenticate using SASL (unknown error)
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Socket connection established to
>> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] WARN
>> org.apache.zookeeper.ClientCnxn
>> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7
>> has
>> >>> expired
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7
>> has
>> >>> expired, closing socket connection
>> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>> >>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>> >>> connectString=qzkpr01.p13.eng.in03.com:50024,
>> >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>> >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>> >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>> >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>> >>> 0x3673a59f5dd01c7
>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Opening socket connection to server
>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>> >>> authenticate using SASL (unknown error)
>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Socket connection established to
>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Session establishment complete on server
>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>> >>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>> >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>> >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>> >>>  at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>>  at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>>  at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>> >>> unmarshal object with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>> >>>  at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>> >>>  ... 10 common frames omitted
>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>> >>> deserialize object with given class loader:
>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>> >>> deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>> >>>  at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>> >>>  ... 16 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>> >>>  ... 18 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 21 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize field
>> >>> [name=assignment]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 27 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>> >>>  ... 30 common frames omitted
>> >>> Caused by: java.io.IOException: Unexpected error occurred during
>> >>> unmarshalling of an instance of the class:
>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> Check
>> >>> that all nodes are running the same version of Ignite and that all
>> nodes
>> >>> have GridOptimizedMarshaller configured with identical optimized
>> classes
>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
>> your
>> >>> serialized classes implement java.io.Externalizable interface, verify
>> that
>> >>> serialization logic is correct.
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 41 common frames omitted
>> >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>> >>>  at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>>  at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>>  at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>> >>> unmarshal object with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>> >>>  at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>> >>>  ... 10 common frames omitted
>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>> >>> deserialize object with given class loader:
>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>> >>> deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>> >>>  at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>> >>>  ... 16 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>> >>>  ... 18 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 21 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize field
>> >>> [name=assignment]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 27 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>> >>>  ... 30 common frames omitted
>> >>> Caused by: java.io.IOException: Unexpected error occurred during
>> >>> unmarshalling of an instance of the class:
>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> Check
>> >>> that all nodes are running the same version of Ignite and that all
>> nodes
>> >>> have GridOptimizedMarshaller configured with identical optimized
>> classes
>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
>> your
>> >>> serialized classes implement java.io.Externalizable interface, verify
>> that
>> >>> serialization logic is correct.
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>>  at
>> >>>
>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>>  ... 41 common frames omitted
>> >>>
>> >>> Thanks,
>> >>> Akash
>> >>>
>> >>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Prasad Bhalerao
In reply to this post by Ilya Kasnacheev


Resending with screenshots.
 

I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.

 

AffinityJob result being serialized on server:

 


 

 

AffinityJob result being de-serialized on client:

 



Thanks,
Prasad


On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Prasad Bhalerao
In reply to this post by Ilya Kasnacheev
Resending....


I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.

 

AffinityJob result being serialized on server:


server.png



AffinityJob result being de-serialized on client:





Thanks,
Prasad

On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Prasad Bhalerao

Resending....


I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.

 

NOTE:  This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].

 

As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.

 

I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.

 

AffinityJob result being serialized on server:


server.png
server.png


AffinityJob result being de-serialized on client:


client.jpg


Thanks,
Prasad

On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

Do you have a reproducer project to reliably confirm this issue?

Regards,
--
Ilya Kasnacheev


ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>:
Added  [hidden email].

Should I log Jira for this issue?

Thanks,
Akash



On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote:

> Hi,
>
> No both nodes, client and server are running on Ignite 2.7 version. I am
> starting both server and client from Intellij IDE.
>
> Version printed in Server node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Version in client node log:
> Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>
> Thanks,
> Akash
>
> On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote:
>
>> Hi!
>>
>> Any chance you might have one node running 2.6 or something like that ?
>>
>> It looks like it get a different object that does not match the one
>> expected in 2.7
>>
>> Mikael
>> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>>
>> Before submitting the affinity task ignite first gets the affinity cached
>> function (AffinityInfo) by submitting the cluster wide task "AffinityJob".
>> But while in the process of retrieving the output of this AffinityJob,
>> ignite deserializes this output. I am getting exception while deserailizing
>> this output.
>> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> CacheMetrics object from input stream on 14th iteration I am getting
>> following exception. Complete stack trace is given in this mail chain.
>>
>> Caused by: java.io.IOException: Unexpected error occurred during
>> unmarshalling of an instance of the class:
>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>>
>> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>>
>> Is this a bug or am I doing something wrong?
>>
>> Can someone please help?
>>
>> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> When execute affinity.partition(key), I am getting following exception
>>> on Ignite  2.7.
>>>
>>> Stacktrace:
>>>
>>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] WARN  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired
>>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>>> qzkpr01.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has
>>> expired, closing socket connection
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>>> connectString=qzkpr01.p13.eng.in03.com:50024,
>>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>>> 0x3673a59f5dd01c7
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Opening socket connection to server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Socket connection established to
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>>> qzkpr03.p13.eng.in03.com:50024)] INFO  org.apache.zookeeper.ClientCnxn
>>> - Session establishment complete on server
>>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>>> isCancelled=false, retry=null]
>>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>>> with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>>>  at
>>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>  at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>  at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>>> unmarshal object with optimized marshaller
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>  at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>  at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>  ... 10 common frames omitted
>>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>>> deserialize object with given class loader:
>>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>>> deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>>>  at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>  at
>>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>>>  ... 16 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>>>  ... 18 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 21 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize field
>>> [name=assignment]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 27 common frames omitted
>>> Caused by: java.io.IOException: Failed to deserialize object
>>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>>>  ... 30 common frames omitted
>>> Caused by: java.io.IOException: Unexpected error occurred during
>>> unmarshalling of an instance of the class:
>>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check
>>> that all nodes are running the same version of Ignite and that all nodes
>>> have GridOptimizedMarshaller configured with identical optimized classes
>>> lists, if any (see setClassNames and setClassNamesPath methods). If your
>>> serialized classes implement java.io.Externalizable interface, verify that
>>> serialization logic is correct.
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>>>  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>>>  at
>>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>>>  at
>>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>>>  ... 41 common frames omitted
>>>
>>> Thanks,
>>> Akash
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Prasad Bhalerao
No I am not using zookeeper discovery.
Using TcpDiscoveryVmIpFinder.

Can someone please explain on what event cacheMetrics in TcpDiscoveryNode
gets populated. It is not getting populated in standalone program.

If it gets populated then I might be able to reproduce this case.

On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev <[hidden email]
wrote:

> Hello!
>
> Have you tried enabling Zookeeper in your reproducer? I have a hunch that
> they are linked: this behavior is affected by zookeeper discovery.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <[hidden email]
> >:
>
>> I tried to reproduce this in standalone program. But the thing is cache
>> metrics map in TcpDiscoveryNode is empty even after setting
>> statisticEnabled to true on all caches.
>> So the flow does not enter into serializr/deserialize cacheMetrics block.
>>
>> Any idea how the cacheMetrics gets populated. On which event?
>>
>>
>> Thanks,
>> Prasad
>>
>> On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email]
>> wrote:
>>
>>> Hello!
>>>
>>> I think the problem was introduced by
>>> https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
>>> suspicious, however it is strange that it does not reproduce right away.
>>>
>>> I could try and devise a fix but I could not reproduce this behavior in
>>> any
>>> of tests. If you could do a reproducer project that would be awesome.
>>>
>>> Regards,
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: [IGNITE-10925] After upgrading 2.7 getting Unexpected error occurred during unmarshalling

Prasad Bhalerao
Hi,

I am able to create a reproducer for this issue. I have also created a JIRA
IGNITE-10925 <https://issues.apache.org/jira/browse/IGNITE-10925>  for this
issue.

Reproducer: https://github.com/prasadbhalerao1983/IgniteIssueReproducer.git

Step to Reproduce:

1) First Run com.example.demo.Server class as a java program

2) Then run com.example.demo.Client as java program.

Thanks,
Prasad

On Sat, Jan 12, 2019 at 11:17 AM Prasad Bhalerao <
[hidden email]> wrote:

> No I am not using zookeeper discovery.
> Using TcpDiscoveryVmIpFinder.
>
> Can someone please explain on what event cacheMetrics in TcpDiscoveryNode
> gets populated. It is not getting populated in standalone program.
>
> If it gets populated then I might be able to reproduce this case.
>
> On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev <[hidden email]
> wrote:
>
>> Hello!
>>
>> Have you tried enabling Zookeeper in your reproducer? I have a hunch that
>> they are linked: this behavior is affected by zookeeper discovery.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <
>> [hidden email]>:
>>
>>> I tried to reproduce this in standalone program. But the thing is cache
>>> metrics map in TcpDiscoveryNode is empty even after setting
>>> statisticEnabled to true on all caches.
>>> So the flow does not enter into serializr/deserialize cacheMetrics block.
>>>
>>> Any idea how the cacheMetrics gets populated. On which event?
>>>
>>>
>>> Thanks,
>>> Prasad
>>>
>>> On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email]
>>> wrote:
>>>
>>>> Hello!
>>>>
>>>> I think the problem was introduced by
>>>> https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
>>>> suspicious, however it is strange that it does not reproduce right away.
>>>>
>>>> I could try and devise a fix but I could not reproduce this behavior in
>>>> any
>>>> of tests. If you could do a reproducer project that would be awesome.
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>