Added [hidden email].
Should I log Jira for this issue? Thanks, Akash On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote: > Hi, > > No both nodes, client and server are running on Ignite 2.7 version. I am > starting both server and client from Intellij IDE. > > Version printed in Server node log: > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db > > Version in client node log: > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db > > Thanks, > Akash > > On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote: > >> Hi! >> >> Any chance you might have one node running 2.6 or something like that ? >> >> It looks like it get a different object that does not match the one >> expected in 2.7 >> >> Mikael >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde: >> >> Before submitting the affinity task ignite first gets the affinity cached >> function (AffinityInfo) by submitting the cluster wide task "AffinityJob". >> But while in the process of retrieving the output of this AffinityJob, >> ignite deserializes this output. I am getting exception while deserailizing >> this output. >> In TcpDiscoveryNode.readExternal() method while deserailizing the >> CacheMetrics object from input stream on 14th iteration I am getting >> following exception. Complete stack trace is given in this mail chain. >> >> Caused by: java.io.IOException: Unexpected error occurred during >> unmarshalling of an instance of the class: >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >> >> This is working fine on Ignite 2.6 version but giving problem on 2.7. >> >> Is this a bug or am I doing something wrong? >> >> Can someone please help? >> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> >> wrote: >> >>> Hi, >>> >>> When execute affinity.partition(key), I am getting following exception >>> on Ignite 2.7. >>> >>> Stacktrace: >>> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >>> isCancelled=false, retry=null] >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >>> with optimized marshaller >>> at >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >>> unmarshal object with optimized marshaller >>> at >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >>> at >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >>> at >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>> at >>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>> at >>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>> at >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>> ... 10 common frames omitted >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >>> deserialize object with given class loader: >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >>> deserialize object >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >>> at >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >>> at >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >>> ... 16 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >>> ... 18 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 21 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize field >>> [name=assignment] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 27 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >>> ... 30 common frames omitted >>> Caused by: java.io.IOException: Unexpected error occurred during >>> unmarshalling of an instance of the class: >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check >>> that all nodes are running the same version of Ignite and that all nodes >>> have GridOptimizedMarshaller configured with identical optimized classes >>> lists, if any (see setClassNames and setClassNamesPath methods). If your >>> serialized classes implement java.io.Externalizable interface, verify that >>> serialization logic is correct. >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 41 common frames omitted >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> qzkpr01.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn >>> - Opening socket connection to server >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to >>> authenticate using SASL (unknown error) >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> qzkpr01.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn >>> - Socket connection established to >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> qzkpr01.p13.eng.in03.com:50024)] WARN org.apache.zookeeper.ClientCnxn >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has >>> expired >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> qzkpr01.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 has >>> expired, closing socket connection >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO >>> org.apache.zookeeper.ZooKeeper - Initiating client connection, >>> connectString=qzkpr01.p13.eng.in03.com:50024, >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288 >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session: >>> 0x3673a59f5dd01c7 >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >>> qzkpr03.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn >>> - Opening socket connection to server >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to >>> authenticate using SASL (unknown error) >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >>> qzkpr03.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn >>> - Socket connection established to >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >>> qzkpr03.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn >>> - Session establishment complete on server >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = >>> 0x3673a59f5dd01ce, negotiated timeout = 10000 >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected) >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >>> isCancelled=false, retry=null] >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >>> with optimized marshaller >>> at >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >>> unmarshal object with optimized marshaller >>> at >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >>> at >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >>> at >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>> at >>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>> at >>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>> at >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>> ... 10 common frames omitted >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >>> deserialize object with given class loader: >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >>> deserialize object >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >>> at >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >>> at >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >>> ... 16 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >>> ... 18 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 21 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize field >>> [name=assignment] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 27 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >>> ... 30 common frames omitted >>> Caused by: java.io.IOException: Unexpected error occurred during >>> unmarshalling of an instance of the class: >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check >>> that all nodes are running the same version of Ignite and that all nodes >>> have GridOptimizedMarshaller configured with identical optimized classes >>> lists, if any (see setClassNames and setClassNamesPath methods). If your >>> serialized classes implement java.io.Externalizable interface, verify that >>> serialization logic is correct. >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 41 common frames omitted >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >>> isCancelled=false, retry=null] >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >>> with optimized marshaller >>> at >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >>> at >>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>> at >>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >>> unmarshal object with optimized marshaller >>> at >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >>> at >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >>> at >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>> at >>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>> at >>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>> at >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>> ... 10 common frames omitted >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >>> deserialize object with given class loader: >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >>> deserialize object >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >>> at >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >>> at >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >>> ... 16 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >>> ... 18 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 21 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize field >>> [name=assignment] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 27 common frames omitted >>> Caused by: java.io.IOException: Failed to deserialize object >>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >>> ... 30 common frames omitted >>> Caused by: java.io.IOException: Unexpected error occurred during >>> unmarshalling of an instance of the class: >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check >>> that all nodes are running the same version of Ignite and that all nodes >>> have GridOptimizedMarshaller configured with identical optimized classes >>> lists, if any (see setClassNames and setClassNamesPath methods). If your >>> serialized classes implement java.io.Externalizable interface, verify that >>> serialization logic is correct. >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> at >>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> at >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> ... 41 common frames omitted >>> >>> Thanks, >>> Akash >>> >> |
Hello!
Do you have a reproducer project to reliably confirm this issue? Regards, -- Ilya Kasnacheev ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>: > Added [hidden email]. > > Should I log Jira for this issue? > > Thanks, > Akash > > > > On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> wrote: > > > Hi, > > > > No both nodes, client and server are running on Ignite 2.7 version. I am > > starting both server and client from Intellij IDE. > > > > Version printed in Server node log: > > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db > > > > Version in client node log: > > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db > > > > Thanks, > > Akash > > > > On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> wrote: > > > >> Hi! > >> > >> Any chance you might have one node running 2.6 or something like that ? > >> > >> It looks like it get a different object that does not match the one > >> expected in 2.7 > >> > >> Mikael > >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde: > >> > >> Before submitting the affinity task ignite first gets the affinity > cached > >> function (AffinityInfo) by submitting the cluster wide task > "AffinityJob". > >> But while in the process of retrieving the output of this AffinityJob, > >> ignite deserializes this output. I am getting exception while > deserailizing > >> this output. > >> In TcpDiscoveryNode.readExternal() method while deserailizing the > >> CacheMetrics object from input stream on 14th iteration I am getting > >> following exception. Complete stack trace is given in this mail chain. > >> > >> Caused by: java.io.IOException: Unexpected error occurred during > >> unmarshalling of an instance of the class: > >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. > >> > >> This is working fine on Ignite 2.6 version but giving problem on 2.7. > >> > >> Is this a bug or am I doing something wrong? > >> > >> Can someone please help? > >> > >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> > >> wrote: > >> > >>> Hi, > >>> > >>> When execute affinity.partition(key), I am getting following exception > >>> on Ignite 2.7. > >>> > >>> Stacktrace: > >>> > >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR > >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: > >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, > >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, > >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, > >>> isCancelled=false, retry=null] > >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object > >>> with optimized marshaller > >>> at > >>> > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>> at java.lang.Thread.run(Thread.java:748) > >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to > >>> unmarshal object with optimized marshaller > >>> at > >>> > org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) > >>> at > >>> > org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) > >>> at > >>> > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) > >>> at > >>> > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) > >>> ... 10 common frames omitted > >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to > >>> deserialize object with given class loader: > >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to > >>> deserialize object > >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) > >>> at > >>> > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) > >>> ... 16 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) > >>> ... 18 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> > [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 21 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize field > >>> [name=assignment] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 27 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> > [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) > >>> ... 30 common frames omitted > >>> Caused by: java.io.IOException: Unexpected error occurred during > >>> unmarshalling of an instance of the class: > >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check > >>> that all nodes are running the same version of Ignite and that all > nodes > >>> have GridOptimizedMarshaller configured with identical optimized > classes > >>> lists, if any (see setClassNames and setClassNamesPath methods). If > your > >>> serialized classes implement java.io.Externalizable interface, verify > that > >>> serialization logic is correct. > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 41 common frames omitted > >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( > >>> qzkpr01.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn > >>> - Opening socket connection to server > >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to > >>> authenticate using SASL (unknown error) > >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( > >>> qzkpr01.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn > >>> - Socket connection established to > >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session > >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( > >>> qzkpr01.p13.eng.in03.com:50024)] WARN org.apache.zookeeper.ClientCnxn > >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 > has > >>> expired > >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( > >>> qzkpr01.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn > >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 > has > >>> expired, closing socket connection > >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO > >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) > >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO > >>> org.apache.zookeeper.ZooKeeper - Initiating client connection, > >>> connectString=qzkpr01.p13.eng.in03.com:50024, > >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka > >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288 > >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO > >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session: > >>> 0x3673a59f5dd01c7 > >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( > >>> qzkpr03.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn > >>> - Opening socket connection to server > >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to > >>> authenticate using SASL (unknown error) > >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( > >>> qzkpr03.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn > >>> - Socket connection established to > >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session > >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( > >>> qzkpr03.p13.eng.in03.com:50024)] INFO org.apache.zookeeper.ClientCnxn > >>> - Session establishment complete on server > >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = > >>> 0x3673a59f5dd01ce, negotiated timeout = 10000 > >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO > >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected) > >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR > >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: > >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, > >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, > >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, > >>> isCancelled=false, retry=null] > >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object > >>> with optimized marshaller > >>> at > >>> > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>> at java.lang.Thread.run(Thread.java:748) > >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to > >>> unmarshal object with optimized marshaller > >>> at > >>> > org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) > >>> at > >>> > org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) > >>> at > >>> > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) > >>> at > >>> > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) > >>> ... 10 common frames omitted > >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to > >>> deserialize object with given class loader: > >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to > >>> deserialize object > >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) > >>> at > >>> > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) > >>> ... 16 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) > >>> ... 18 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> > [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 21 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize field > >>> [name=assignment] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 27 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> > [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) > >>> ... 30 common frames omitted > >>> Caused by: java.io.IOException: Unexpected error occurred during > >>> unmarshalling of an instance of the class: > >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check > >>> that all nodes are running the same version of Ignite and that all > nodes > >>> have GridOptimizedMarshaller configured with identical optimized > classes > >>> lists, if any (see setClassNames and setClassNamesPath methods). If > your > >>> serialized classes implement java.io.Externalizable interface, verify > that > >>> serialization logic is correct. > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 41 common frames omitted > >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR > >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: > >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, > >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, > >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, > >>> isCancelled=false, retry=null] > >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object > >>> with optimized marshaller > >>> at > >>> > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) > >>> at > >>> > org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > >>> at > >>> > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>> at > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>> at java.lang.Thread.run(Thread.java:748) > >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to > >>> unmarshal object with optimized marshaller > >>> at > >>> > org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) > >>> at > >>> > org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) > >>> at > >>> > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) > >>> at > >>> > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) > >>> ... 10 common frames omitted > >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to > >>> deserialize object with given class loader: > >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to > >>> deserialize object > >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) > >>> at > >>> > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) > >>> at > >>> > org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) > >>> ... 16 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) > >>> ... 18 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> > [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 21 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize field > >>> [name=assignment] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 27 common frames omitted > >>> Caused by: java.io.IOException: Failed to deserialize object > >>> > [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) > >>> ... 30 common frames omitted > >>> Caused by: java.io.IOException: Unexpected error occurred during > >>> unmarshalling of an instance of the class: > >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check > >>> that all nodes are running the same version of Ignite and that all > nodes > >>> have GridOptimizedMarshaller configured with identical optimized > classes > >>> lists, if any (see setClassNames and setClassNamesPath methods). If > your > >>> serialized classes implement java.io.Externalizable interface, verify > that > >>> serialization logic is correct. > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) > >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) > >>> at > >>> > org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) > >>> at > >>> > org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) > >>> ... 41 common frames omitted > >>> > >>> Thanks, > >>> Akash > >>> > >> > |
Hi Ilya,
I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail. *NOTE: *This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)]. As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics gets serialized successfully on server node. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception. I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 version all 22 cacheMetrics are being de-serialized successfully. This Looks like a bug to me in serializer/deseriliazer code. *AffinityJob result being serialized on server:* *AffinityJob result being de-serialized on client:* Thanks, Prasad On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote: > Hello! > > Do you have a reproducer project to reliably confirm this issue? > > Regards, > -- > Ilya Kasnacheev > > > ср, 9 янв. 2019 г. в 12:39, Akash Shinde <[hidden email]>: > >> Added [hidden email]. >> >> Should I log Jira for this issue? >> >> Thanks, >> Akash >> >> >> >> On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <[hidden email]> >> wrote: >> >> > Hi, >> > >> > No both nodes, client and server are running on Ignite 2.7 version. I am >> > starting both server and client from Intellij IDE. >> > >> > Version printed in Server node log: >> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db >> > >> > Version in client node log: >> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db >> > >> > Thanks, >> > Akash >> > >> > On Tue, Jan 8, 2019 at 5:18 PM Mikael <[hidden email]> >> wrote: >> > >> >> Hi! >> >> >> >> Any chance you might have one node running 2.6 or something like that ? >> >> >> >> It looks like it get a different object that does not match the one >> >> expected in 2.7 >> >> >> >> Mikael >> >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde: >> >> >> >> Before submitting the affinity task ignite first gets the affinity >> cached >> >> function (AffinityInfo) by submitting the cluster wide task >> "AffinityJob". >> >> But while in the process of retrieving the output of this AffinityJob, >> >> ignite deserializes this output. I am getting exception while >> deserailizing >> >> this output. >> >> In TcpDiscoveryNode.readExternal() method while deserailizing the >> >> CacheMetrics object from input stream on 14th iteration I am getting >> >> following exception. Complete stack trace is given in this mail chain. >> >> >> >> Caused by: java.io.IOException: Unexpected error occurred during >> >> unmarshalling of an instance of the class: >> >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >> >> >> >> This is working fine on Ignite 2.6 version but giving problem on 2.7. >> >> >> >> Is this a bug or am I doing something wrong? >> >> >> >> Can someone please help? >> >> >> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <[hidden email]> >> >> wrote: >> >> >> >>> Hi, >> >>> >> >>> When execute affinity.partition(key), I am getting following exception >> >>> on Ignite 2.7. >> >>> >> >>> Stacktrace: >> >>> >> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR >> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >> >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >> >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >> >>> isCancelled=false, retry=null] >> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >> >>> with optimized marshaller >> >>> at >> >>> >> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >>> at java.lang.Thread.run(Thread.java:748) >> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >> >>> unmarshal object with optimized marshaller >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >> >>> at >> >>> >> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >> >>> at >> >>> >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >> >>> at >> >>> >> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >> >>> ... 10 common frames omitted >> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >> >>> deserialize object with given class loader: >> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >> >>> deserialize object >> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >> >>> at >> >>> >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >> >>> ... 16 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >> >>> ... 18 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> >> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 21 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize field >> >>> [name=assignment] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 27 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> >> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >> >>> ... 30 common frames omitted >> >>> Caused by: java.io.IOException: Unexpected error occurred during >> >>> unmarshalling of an instance of the class: >> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >> Check >> >>> that all nodes are running the same version of Ignite and that all >> nodes >> >>> have GridOptimizedMarshaller configured with identical optimized >> classes >> >>> lists, if any (see setClassNames and setClassNamesPath methods). If >> your >> >>> serialized classes implement java.io.Externalizable interface, verify >> that >> >>> serialization logic is correct. >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 41 common frames omitted >> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >> >>> qzkpr01.p13.eng.in03.com:50024)] INFO >> org.apache.zookeeper.ClientCnxn >> >>> - Opening socket connection to server >> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to >> >>> authenticate using SASL (unknown error) >> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >> >>> qzkpr01.p13.eng.in03.com:50024)] INFO >> org.apache.zookeeper.ClientCnxn >> >>> - Socket connection established to >> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session >> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >> >>> qzkpr01.p13.eng.in03.com:50024)] WARN >> org.apache.zookeeper.ClientCnxn >> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 >> has >> >>> expired >> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >> >>> qzkpr01.p13.eng.in03.com:50024)] INFO >> org.apache.zookeeper.ClientCnxn >> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7 >> has >> >>> expired, closing socket connection >> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO >> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) >> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO >> >>> org.apache.zookeeper.ZooKeeper - Initiating client connection, >> >>> connectString=qzkpr01.p13.eng.in03.com:50024, >> >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka >> >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288 >> >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO >> >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session: >> >>> 0x3673a59f5dd01c7 >> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >> >>> qzkpr03.p13.eng.in03.com:50024)] INFO >> org.apache.zookeeper.ClientCnxn >> >>> - Opening socket connection to server >> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to >> >>> authenticate using SASL (unknown error) >> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >> >>> qzkpr03.p13.eng.in03.com:50024)] INFO >> org.apache.zookeeper.ClientCnxn >> >>> - Socket connection established to >> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session >> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >> >>> qzkpr03.p13.eng.in03.com:50024)] INFO >> org.apache.zookeeper.ClientCnxn >> >>> - Session establishment complete on server >> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = >> >>> 0x3673a59f5dd01ce, negotiated timeout = 10000 >> >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO >> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected) >> >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR >> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >> >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >> >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >> >>> isCancelled=false, retry=null] >> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >> >>> with optimized marshaller >> >>> at >> >>> >> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >>> at java.lang.Thread.run(Thread.java:748) >> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >> >>> unmarshal object with optimized marshaller >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >> >>> at >> >>> >> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >> >>> at >> >>> >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >> >>> at >> >>> >> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >> >>> ... 10 common frames omitted >> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >> >>> deserialize object with given class loader: >> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >> >>> deserialize object >> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >> >>> at >> >>> >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >> >>> ... 16 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >> >>> ... 18 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> >> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 21 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize field >> >>> [name=assignment] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 27 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> >> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >> >>> ... 30 common frames omitted >> >>> Caused by: java.io.IOException: Unexpected error occurred during >> >>> unmarshalling of an instance of the class: >> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >> Check >> >>> that all nodes are running the same version of Ignite and that all >> nodes >> >>> have GridOptimizedMarshaller configured with identical optimized >> classes >> >>> lists, if any (see setClassNames and setClassNamesPath methods). If >> your >> >>> serialized classes implement java.io.Externalizable interface, verify >> that >> >>> serialization logic is correct. >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 41 common frames omitted >> >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR >> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >> >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >> >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >> >>> isCancelled=false, retry=null] >> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >> >>> with optimized marshaller >> >>> at >> >>> >> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >> >>> at >> >>> >> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >> >>> at >> >>> >> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >>> at java.lang.Thread.run(Thread.java:748) >> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >> >>> unmarshal object with optimized marshaller >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >> >>> at >> >>> >> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >> >>> at >> >>> >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >> >>> at >> >>> >> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >> >>> ... 10 common frames omitted >> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >> >>> deserialize object with given class loader: >> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >> >>> deserialize object >> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >> >>> at >> >>> >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >> >>> at >> >>> >> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >> >>> ... 16 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >> >>> ... 18 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> >> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 21 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize field >> >>> [name=assignment] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 27 common frames omitted >> >>> Caused by: java.io.IOException: Failed to deserialize object >> >>> >> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >> >>> ... 30 common frames omitted >> >>> Caused by: java.io.IOException: Unexpected error occurred during >> >>> unmarshalling of an instance of the class: >> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >> Check >> >>> that all nodes are running the same version of Ignite and that all >> nodes >> >>> have GridOptimizedMarshaller configured with identical optimized >> classes >> >>> lists, if any (see setClassNames and setClassNamesPath methods). If >> your >> >>> serialized classes implement java.io.Externalizable interface, verify >> that >> >>> serialization logic is correct. >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >> >>> at >> >>> >> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >> >>> at >> >>> >> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >> >>> ... 41 common frames omitted >> >>> >> >>> Thanks, >> >>> Akash >> >>> >> >> >> > |
In reply to this post by Ilya Kasnacheev
Resending with screenshots.
|
In reply to this post by Ilya Kasnacheev
Resending.... I am not able reproduce this issue in small reproducer project but this is consistently happening in my project. So I debugged this issue and attached the screenshot in this mail.
NOTE: This issue occurs if the statistics are enabled on cache configuration level [cacheCfg.setStatisticsEnabled(true)].
As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All these cache metrics get serialized successfully. But at the time of deserialization on client node only first metrics get de-serialized successfully but all other metrics till iteration count 13 are de-serialized as null value and on iteration 14 “ref” byte value in “OptimizedObjectInputStream.readObject0()” method is read as 81 and code throws exception.
I think this where it is going wrong. The object copy at the time of serialization and de-serialization should be same but that’s not happening in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all 22 cacheMetrics are being de-serialized successfully.
AffinityJob result being serialized on server: AffinityJob result being de-serialized on client: Thanks, Prasad On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <[hidden email]> wrote:
|
|
No I am not using zookeeper discovery.
Using TcpDiscoveryVmIpFinder. Can someone please explain on what event cacheMetrics in TcpDiscoveryNode gets populated. It is not getting populated in standalone program. If it gets populated then I might be able to reproduce this case. On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev <[hidden email] wrote: > Hello! > > Have you tried enabling Zookeeper in your reproducer? I have a hunch that > they are linked: this behavior is affected by zookeeper discovery. > > Regards, > -- > Ilya Kasnacheev > > > пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <[hidden email] > >: > >> I tried to reproduce this in standalone program. But the thing is cache >> metrics map in TcpDiscoveryNode is empty even after setting >> statisticEnabled to true on all caches. >> So the flow does not enter into serializr/deserialize cacheMetrics block. >> >> Any idea how the cacheMetrics gets populated. On which event? >> >> >> Thanks, >> Prasad >> >> On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email] >> wrote: >> >>> Hello! >>> >>> I think the problem was introduced by >>> https://issues.apache.org/jira/browse/IGNITE-6846 which does look very >>> suspicious, however it is strange that it does not reproduce right away. >>> >>> I could try and devise a fix but I could not reproduce this behavior in >>> any >>> of tests. If you could do a reproducer project that would be awesome. >>> >>> Regards, >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >> |
Hi,
I am able to create a reproducer for this issue. I have also created a JIRA IGNITE-10925 <https://issues.apache.org/jira/browse/IGNITE-10925> for this issue. Reproducer: https://github.com/prasadbhalerao1983/IgniteIssueReproducer.git Step to Reproduce: 1) First Run com.example.demo.Server class as a java program 2) Then run com.example.demo.Client as java program. Thanks, Prasad On Sat, Jan 12, 2019 at 11:17 AM Prasad Bhalerao < [hidden email]> wrote: > No I am not using zookeeper discovery. > Using TcpDiscoveryVmIpFinder. > > Can someone please explain on what event cacheMetrics in TcpDiscoveryNode > gets populated. It is not getting populated in standalone program. > > If it gets populated then I might be able to reproduce this case. > > On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev <[hidden email] > wrote: > >> Hello! >> >> Have you tried enabling Zookeeper in your reproducer? I have a hunch that >> they are linked: this behavior is affected by zookeeper discovery. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao < >> [hidden email]>: >> >>> I tried to reproduce this in standalone program. But the thing is cache >>> metrics map in TcpDiscoveryNode is empty even after setting >>> statisticEnabled to true on all caches. >>> So the flow does not enter into serializr/deserialize cacheMetrics block. >>> >>> Any idea how the cacheMetrics gets populated. On which event? >>> >>> >>> Thanks, >>> Prasad >>> >>> On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev <[hidden email] >>> wrote: >>> >>>> Hello! >>>> >>>> I think the problem was introduced by >>>> https://issues.apache.org/jira/browse/IGNITE-6846 which does look very >>>> suspicious, however it is strange that it does not reproduce right away. >>>> >>>> I could try and devise a fix but I could not reproduce this behavior in >>>> any >>>> of tests. If you could do a reproducer project that would be awesome. >>>> >>>> Regards, >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>>> >>> |
Free forum by Nabble | Edit this page |