Apache Ignite Developers - Legacy Mail Archive

Cluster group affinity

Classic

List

Threaded

14 messages Options

Andrey Kornev

Cluster group affinity

Hello,

I have a user-defined cluster group and I'd like to be able to consistently pick the same node in the group for a given key. Essentially, what I want is a cluster group affinity that is not associated with any cache. How can I do it?

Thanks
Andrey

dsetrakyan

Re: Cluster group affinity

On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <[hidden email]>
wrote:

> Hello,
>
> I have a user-defined cluster group and I'd like to be able to
> consistently pick the same node in the group for a given key. Essentially,
> what I want is a cluster group affinity that is not associated with any
> cache. How can I do it?
>

Andrey, perhaps you could just take our affinity function and use it
directly, no?

>
> Thanks
> Andrey
>

Andrey Kornev

RE: Cluster group affinity

Dmitriy,

The affinity function only maps a key to a partition id and it doesn't seem to provide a way to map the partition id to a cluster node. So I'm a little bit confused right now.

Could you please clarify?

Thanks a lot
Andrey

> From: [hidden email]
> Date: Mon, 5 Oct 2015 09:53:25 -0700
> Subject: Re: Cluster group affinity
> To: [hidden email]
>
> On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <[hidden email]>
> wrote:
>
> > Hello,
> >
> > I have a user-defined cluster group and I'd like to be able to
> > consistently pick the same node in the group for a given key. Essentially,
> > what I want is a cluster group affinity that is not associated with any
> > cache. How can I do it?
> >
>
> Andrey, perhaps you could just take our affinity function and use it
> directly, no?
>
>
> >
> > Thanks
> > Andrey
> >

Andrey Gura

Re: Cluster group affinity

Andrey,

See AffinityFunction.assignPartitions method. It returns assignment list as
List<List<ClusterNode>> where index of element in returned list corresponds
to partition number. Assignment for each partition represented as list of
nodes where primary node is always the first. So you can use existing
affinity functions for you case just passing properly formed
AffinityFunctionContext to assignPartitions method.

On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <[hidden email]>
wrote:

> Dmitriy,
>
> The affinity function only maps a key to a partition id and it doesn't
> seem to provide a way to map the partition id to a cluster node. So I'm a
> little bit confused right now.
>
> Could you please clarify?
>
> Thanks a lot
> Andrey
>
> > From: [hidden email]
> > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > Subject: Re: Cluster group affinity
> > To: [hidden email]
> >
> > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Hello,
> > >
> > > I have a user-defined cluster group and I'd like to be able to
> > > consistently pick the same node in the group for a given key.
> Essentially,
> > > what I want is a cluster group affinity that is not associated with any
> > > cache. How can I do it?
> > >
> >
> > Andrey, perhaps you could just take our affinity function and use it
> > directly, no?
> >
> >
> > >
> > > Thanks
> > > Andrey
> > >
>
>

--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Andrey Kornev

RE: Cluster group affinity

Andrey, thanks!

But a "properly formed AffinityFunctionContext" is the problem:
1) I'm expected to return an instance of the internal class AffinityTopologyVersion.
2) the consequences of returning null from AffinityFunctionContext.previousAssignment and AffinityFunctionContext.discoveryEvent methods (because I can't provide any meaningful implementation for them) are not clear.

Please advise.

Thanks
Andrey

> Date: Tue, 6 Oct 2015 16:43:10 +0300
> Subject: Re: Cluster group affinity
> From: [hidden email]
> To: [hidden email]
>
> Andrey,
>
> See AffinityFunction.assignPartitions method. It returns assignment list as
> List<List<ClusterNode>> where index of element in returned list corresponds
> to partition number. Assignment for each partition represented as list of
> nodes where primary node is always the first. So you can use existing
> affinity functions for you case just passing properly formed
> AffinityFunctionContext to assignPartitions method.
>
> On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <[hidden email]>
> wrote:
>
> > Dmitriy,
> >
> > The affinity function only maps a key to a partition id and it doesn't
> > seem to provide a way to map the partition id to a cluster node. So I'm a
> > little bit confused right now.
> >
> > Could you please clarify?
> >
> > Thanks a lot
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > Subject: Re: Cluster group affinity
> > > To: [hidden email]
> > >
> > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <[hidden email]>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I have a user-defined cluster group and I'd like to be able to
> > > > consistently pick the same node in the group for a given key.
> > Essentially,
> > > > what I want is a cluster group affinity that is not associated with any
> > > > cache. How can I do it?
> > > >
> > >
> > > Andrey, perhaps you could just take our affinity function and use it
> > > directly, no?
> > >
> > >
> > > >
> > > > Thanks
> > > > Andrey
> > > >
> >
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com

Andrey Gura

Re: Cluster group affinity

Andrey,

> 1) I'm expected to return an instance of the internal class
> AffinityTopologyVersion.

If you are talking about AffinityContextFunction.currentTopologyVersion
method then for now this method is nowhere uses. But it make sense to
return non null value in order to avoid problems in the future.

2) the consequences of returning null from
> AffinityFunctionContext.previousAssignment and
> AffinityFunctionContext.discoveryEvent methods (because I can't provide any
> meaningful implementation for them) are not clear.
>

Both methods declared as @Nullable, so affinity function developer should
correctly handle this cases. In Ignite only FairAffinityFunction uses these
methods. FairAffinityFunction tries to obtain left node Id from event of
EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs to
exclude this node assignment from previous assignments. So if your cluster
group lost node you can return EVT_NODE_LEFT discovery event with Id of
lost node from discoveryEvent method and assignments for previous cluster
group state from previousAssignment method.

RendezvousAffinityFunction uses only currentTopologySnapshot() and
backups() methods of AffinityFunctionContext interface.

On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <[hidden email]>
wrote:

> Andrey, thanks!
>
> But a "properly formed AffinityFunctionContext" is the problem:
> 1) I'm expected to return an instance of the internal class
> AffinityTopologyVersion.
> 2) the consequences of returning null from
> AffinityFunctionContext.previousAssignment and
> AffinityFunctionContext.discoveryEvent methods (because I can't provide any
> meaningful implementation for them) are not clear.
>
> Please advise.
>
> Thanks
> Andrey
>
> > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > Subject: Re: Cluster group affinity
> > From: [hidden email]
> > To: [hidden email]
> >
> > Andrey,
> >
> > See AffinityFunction.assignPartitions method. It returns assignment list
> as
> > List<List<ClusterNode>> where index of element in returned list
> corresponds
> > to partition number. Assignment for each partition represented as list of
> > nodes where primary node is always the first. So you can use existing
> > affinity functions for you case just passing properly formed
> > AffinityFunctionContext to assignPartitions method.
> >
> > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > The affinity function only maps a key to a partition id and it doesn't
> > > seem to provide a way to map the partition id to a cluster node. So
> I'm a
> > > little bit confused right now.
> > >
> > > Could you please clarify?
> > >
> > > Thanks a lot
> > > Andrey
> > >
> > > > From: [hidden email]
> > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > Subject: Re: Cluster group affinity
> > > > To: [hidden email]
> > > >
> > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I have a user-defined cluster group and I'd like to be able to
> > > > > consistently pick the same node in the group for a given key.
> > > Essentially,
> > > > > what I want is a cluster group affinity that is not associated
> with any
> > > > > cache. How can I do it?
> > > > >
> > > >
> > > > Andrey, perhaps you could just take our affinity function and use it
> > > > directly, no?
> > > >
> > > >
> > > > >
> > > > > Thanks
> > > > > Andrey
> > > > >
> > >
> > >
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
>
>

--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Andrey Kornev

RE: Cluster group affinity

Thanks, Andrey! This definitely helps.

It's just that implementing such a simple feature in the "user space" feels awkward and requires intimate knowledge of fairly low-level details of how things work in the current version.

Just curios, how about providing an override for Ignite.affinity() method that ClusterGroup? Is there something fundamentally wrong about calculating the affinity for an arbitrary collection of nodes (such as a ClusterGroup is)?

Regards
Andrey

> Date: Tue, 6 Oct 2015 18:12:48 +0300
> Subject: Re: Cluster group affinity
> From: [hidden email]
> To: [hidden email]
>
> Andrey,
>
>
> > 1) I'm expected to return an instance of the internal class
> > AffinityTopologyVersion.
>
>
> If you are talking about AffinityContextFunction.currentTopologyVersion
> method then for now this method is nowhere uses. But it make sense to
> return non null value in order to avoid problems in the future.
>
> 2) the consequences of returning null from
> > AffinityFunctionContext.previousAssignment and
> > AffinityFunctionContext.discoveryEvent methods (because I can't provide any
> > meaningful implementation for them) are not clear.
> >
>
> Both methods declared as @Nullable, so affinity function developer should
> correctly handle this cases. In Ignite only FairAffinityFunction uses these
> methods. FairAffinityFunction tries to obtain left node Id from event of
> EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs to
> exclude this node assignment from previous assignments. So if your cluster
> group lost node you can return EVT_NODE_LEFT discovery event with Id of
> lost node from discoveryEvent method and assignments for previous cluster
> group state from previousAssignment method.
>
> RendezvousAffinityFunction uses only currentTopologySnapshot() and
> backups() methods of AffinityFunctionContext interface.
>
>
> On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <[hidden email]>
> wrote:
>
> > Andrey, thanks!
> >
> > But a "properly formed AffinityFunctionContext" is the problem:
> > 1) I'm expected to return an instance of the internal class
> > AffinityTopologyVersion.
> > 2) the consequences of returning null from
> > AffinityFunctionContext.previousAssignment and
> > AffinityFunctionContext.discoveryEvent methods (because I can't provide any
> > meaningful implementation for them) are not clear.
> >
> > Please advise.
> >
> > Thanks
> > Andrey
> >
> > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > Subject: Re: Cluster group affinity
> > > From: [hidden email]
> > > To: [hidden email]
> > >
> > > Andrey,
> > >
> > > See AffinityFunction.assignPartitions method. It returns assignment list
> > as
> > > List<List<ClusterNode>> where index of element in returned list
> > corresponds
> > > to partition number. Assignment for each partition represented as list of
> > > nodes where primary node is always the first. So you can use existing
> > > affinity functions for you case just passing properly formed
> > > AffinityFunctionContext to assignPartitions method.
> > >
> > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <[hidden email]>
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > The affinity function only maps a key to a partition id and it doesn't
> > > > seem to provide a way to map the partition id to a cluster node. So
> > I'm a
> > > > little bit confused right now.
> > > >
> > > > Could you please clarify?
> > > >
> > > > Thanks a lot
> > > > Andrey
> > > >
> > > > > From: [hidden email]
> > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > Subject: Re: Cluster group affinity
> > > > > To: [hidden email]
> > > > >
> > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I have a user-defined cluster group and I'd like to be able to
> > > > > > consistently pick the same node in the group for a given key.
> > > > Essentially,
> > > > > > what I want is a cluster group affinity that is not associated
> > with any
> > > > > > cache. How can I do it?
> > > > > >
> > > > >
> > > > > Andrey, perhaps you could just take our affinity function and use it
> > > > > directly, no?
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Andrey
> > > > > >
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Andrey Gura
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> >
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com

dsetrakyan

Re: Cluster group affinity

On Tue, Oct 6, 2015 at 8:46 AM, Andrey Kornev <[hidden email]>
wrote:

Affinity is usually associated with data. In your case you have no data,
but you still need keys to be always mapped to the same node. How about
creating an empty cache and using standard cache API for determining the
affinity for a key?

> Regards
> Andrey
>
> > Date: Tue, 6 Oct 2015 18:12:48 +0300
> > Subject: Re: Cluster group affinity
> > From: [hidden email]
> > To: [hidden email]
> >
> > Andrey,
> >
> >
> > > 1) I'm expected to return an instance of the internal class
> > > AffinityTopologyVersion.
> >
> >
> > If you are talking about AffinityContextFunction.currentTopologyVersion
> > method then for now this method is nowhere uses. But it make sense to
> > return non null value in order to avoid problems in the future.
> >
> > 2) the consequences of returning null from
> > > AffinityFunctionContext.previousAssignment and
> > > AffinityFunctionContext.discoveryEvent methods (because I can't
> provide any
> > > meaningful implementation for them) are not clear.
> > >
> >
> > Both methods declared as @Nullable, so affinity function developer should
> > correctly handle this cases. In Ignite only FairAffinityFunction uses
> these
> > methods. FairAffinityFunction tries to obtain left node Id from event of
> > EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs to
> > exclude this node assignment from previous assignments. So if your
> cluster
> > group lost node you can return EVT_NODE_LEFT discovery event with Id of
> > lost node from discoveryEvent method and assignments for previous cluster
> > group state from previousAssignment method.
> >
> > RendezvousAffinityFunction uses only currentTopologySnapshot() and
> > backups() methods of AffinityFunctionContext interface.
> >
> >
> > On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Andrey, thanks!
> > >
> > > But a "properly formed AffinityFunctionContext" is the problem:
> > > 1) I'm expected to return an instance of the internal class
> > > AffinityTopologyVersion.
> > > 2) the consequences of returning null from
> > > AffinityFunctionContext.previousAssignment and
> > > AffinityFunctionContext.discoveryEvent methods (because I can't
> provide any
> > > meaningful implementation for them) are not clear.
> > >
> > > Please advise.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > > Subject: Re: Cluster group affinity
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Andrey,
> > > >
> > > > See AffinityFunction.assignPartitions method. It returns assignment
> list
> > > as
> > > > List<List<ClusterNode>> where index of element in returned list
> > > corresponds
> > > > to partition number. Assignment for each partition represented as
> list of
> > > > nodes where primary node is always the first. So you can use existing
> > > > affinity functions for you case just passing properly formed
> > > > AffinityFunctionContext to assignPartitions method.
> > > >
> > > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > The affinity function only maps a key to a partition id and it
> doesn't
> > > > > seem to provide a way to map the partition id to a cluster node. So
> > > I'm a
> > > > > little bit confused right now.
> > > > >
> > > > > Could you please clarify?
> > > > >
> > > > > Thanks a lot
> > > > > Andrey
> > > > >
> > > > > > From: [hidden email]
> > > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > > Subject: Re: Cluster group affinity
> > > > > > To: [hidden email]
> > > > > >
> > > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I have a user-defined cluster group and I'd like to be able to
> > > > > > > consistently pick the same node in the group for a given key.
> > > > > Essentially,
> > > > > > > what I want is a cluster group affinity that is not associated
> > > with any
> > > > > > > cache. How can I do it?
> > > > > > >
> > > > > >
> > > > > > Andrey, perhaps you could just take our affinity function and
> use it
> > > > > > directly, no?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Andrey
> > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Andrey Gura
> > > > GridGain Systems, Inc.
> > > > www.gridgain.com
> > >
> > >
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
>
>

Andrey Gura

Re: Cluster group affinity

In reply to this post by Andrey Kornev

>
> Just curios, how about providing an override for Ignite.affinity() method
> that ClusterGroup? Is there something fundamentally wrong about calculating
> the affinity for an arbitrary collection of nodes (such as a ClusterGroup
> is)?
>

Andrey,

I think Ignite.affinity() method for ClusterGroup does not make sense in
API because for each cache topology snapshot always consist of cache
affinity nodes. However, I don't see any fundamentally problems in using
affinity fucntion for some collection of nodes. But you should make sure
that your code is aware of changes in this collection.

On Tue, Oct 6, 2015 at 6:46 PM, Andrey Kornev <[hidden email]>
wrote:

> Thanks, Andrey! This definitely helps.
>
> It's just that implementing such a simple feature in the "user space"
> feels awkward and requires intimate knowledge of fairly low-level details
> of how things work in the current version.
>
> Just curios, how about providing an override for Ignite.affinity() method
> that ClusterGroup? Is there something fundamentally wrong about calculating
> the affinity for an arbitrary collection of nodes (such as a ClusterGroup
> is)?
>
> Regards
> Andrey
>
> > Date: Tue, 6 Oct 2015 18:12:48 +0300
> > Subject: Re: Cluster group affinity
> > From: [hidden email]
> > To: [hidden email]
> >
> > Andrey,
> >
> >
> > > 1) I'm expected to return an instance of the internal class
> > > AffinityTopologyVersion.
> >
> >
> > If you are talking about AffinityContextFunction.currentTopologyVersion
> > method then for now this method is nowhere uses. But it make sense to
> > return non null value in order to avoid problems in the future.
> >
> > 2) the consequences of returning null from
> > > AffinityFunctionContext.previousAssignment and
> > > AffinityFunctionContext.discoveryEvent methods (because I can't
> provide any
> > > meaningful implementation for them) are not clear.
> > >
> >
> > Both methods declared as @Nullable, so affinity function developer should
> > correctly handle this cases. In Ignite only FairAffinityFunction uses
> these
> > methods. FairAffinityFunction tries to obtain left node Id from event of
> > EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs to
> > exclude this node assignment from previous assignments. So if your
> cluster
> > group lost node you can return EVT_NODE_LEFT discovery event with Id of
> > lost node from discoveryEvent method and assignments for previous cluster
> > group state from previousAssignment method.
> >
> > RendezvousAffinityFunction uses only currentTopologySnapshot() and
> > backups() methods of AffinityFunctionContext interface.
> >
> >
> > On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Andrey, thanks!
> > >
> > > But a "properly formed AffinityFunctionContext" is the problem:
> > > 1) I'm expected to return an instance of the internal class
> > > AffinityTopologyVersion.
> > > 2) the consequences of returning null from
> > > AffinityFunctionContext.previousAssignment and
> > > AffinityFunctionContext.discoveryEvent methods (because I can't
> provide any
> > > meaningful implementation for them) are not clear.
> > >
> > > Please advise.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > > Subject: Re: Cluster group affinity
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Andrey,
> > > >
> > > > See AffinityFunction.assignPartitions method. It returns assignment
> list
> > > as
> > > > List<List<ClusterNode>> where index of element in returned list
> > > corresponds
> > > > to partition number. Assignment for each partition represented as
> list of
> > > > nodes where primary node is always the first. So you can use existing
> > > > affinity functions for you case just passing properly formed
> > > > AffinityFunctionContext to assignPartitions method.
> > > >
> > > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > The affinity function only maps a key to a partition id and it
> doesn't
> > > > > seem to provide a way to map the partition id to a cluster node. So
> > > I'm a
> > > > > little bit confused right now.
> > > > >
> > > > > Could you please clarify?
> > > > >
> > > > > Thanks a lot
> > > > > Andrey
> > > > >
> > > > > > From: [hidden email]
> > > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > > Subject: Re: Cluster group affinity
> > > > > > To: [hidden email]
> > > > > >
> > > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I have a user-defined cluster group and I'd like to be able to
> > > > > > > consistently pick the same node in the group for a given key.
> > > > > Essentially,
> > > > > > > what I want is a cluster group affinity that is not associated
> > > with any
> > > > > > > cache. How can I do it?
> > > > > > >
> > > > > >
> > > > > > Andrey, perhaps you could just take our affinity function and
> use it
> > > > > > directly, no?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Andrey
> > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Andrey Gura
> > > > GridGain Systems, Inc.
> > > > www.gridgain.com
> > >
> > >
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
>
>

--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Andrey Kornev

Re: Cluster group affinity

Maybe I don't quite get your point, Andrey, but to me, the cache affinity nodes could just be equally expressed as a ClusterGroup.forCache(...) cluster group. Then, why can't Ignite expose just a single method that takes a cluster group (rather than a cache name) as a parameter and returns an Affinity instance corresponding to the group? Cheers!
_____________________________
From: Andrey Gura <[hidden email]>
Sent: Tuesday, October 6, 2015 9:31 PM
Subject: Re: Cluster group affinity
To: <[hidden email]>

>
> Just curios, how about providing an override for Ignite.affinity() method
> that ClusterGroup? Is there something fundamentally wrong about calculating
> the affinity for an arbitrary collection of nodes (such as a ClusterGroup
> is)?
>

Andrey,

I think Ignite.affinity() method for ClusterGroup does not make sense in
API because for each cache topology snapshot always consist of cache
affinity nodes. However, I don't see any fundamentally problems in using
affinity fucntion for some collection of nodes. But you should make sure
that your code is aware of changes in this collection.

On Tue, Oct 6, 2015 at 6:46 PM, Andrey Kornev <[hidden email]>
wrote:

> Thanks, Andrey! This definitely helps.
>
> It's just that implementing such a simple feature in the "user space"
> feels awkward and requires intimate knowledge of fairly low-level details
> of how things work in the current version.
>
> Just curios, how about providing an override for Ignite.affinity() method
> that ClusterGroup? Is there something fundamentally wrong about calculating
> the affinity for an arbitrary collection of nodes (such as a ClusterGroup
> is)?
>
> Regards
> Andrey
>
> > Date: Tue, 6 Oct 2015 18:12:48 +0300
> > Subject: Re: Cluster group affinity
> > From: [hidden email]
> > To: [hidden email]
> >
> > Andrey,
> >
> >
> > > 1) I'm expected to return an instance of the internal class
> > > AffinityTopologyVersion.
> >
> >
> > If you are talking about AffinityContextFunction.currentTopologyVersion
> > method then for now this method is nowhere uses. But it make sense to
> > return non null value in order to avoid problems in the future.
> >
> > 2) the consequences of returning null from
> > > AffinityFunctionContext.previousAssignment and
> > > AffinityFunctionContext.discoveryEvent methods (because I can't
> provide any
> > > meaningful implementation for them) are not clear.
> > >
> >
> > Both methods declared as @Nullable, so affinity function developer should
> > correctly handle this cases. In Ignite only FairAffinityFunction uses
> these
> > methods. FairAffinityFunction tries to obtain left node Id from event of
> > EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs to
> > exclude this node assignment from previous assignments. So if your
> cluster
> > group lost node you can return EVT_NODE_LEFT discovery event with Id of
> > lost node from discoveryEvent method and assignments for previous cluster
> > group state from previousAssignment method.
> >
> > RendezvousAffinityFunction uses only currentTopologySnapshot() and
> > backups() methods of AffinityFunctionContext interface.
> >
> >
> > On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Andrey, thanks!
> > >
> > > But a "properly formed AffinityFunctionContext" is the problem:
> > > 1) I'm expected to return an instance of the internal class
> > > AffinityTopologyVersion.
> > > 2) the consequences of returning null from
> > > AffinityFunctionContext.previousAssignment and
> > > AffinityFunctionContext.discoveryEvent methods (because I can't
> provide any
> > > meaningful implementation for them) are not clear.
> > >
> > > Please advise.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > > Subject: Re: Cluster group affinity
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Andrey,
> > > >
> > > > See AffinityFunction.assignPartitions method. It returns assignment
> list
> > > as
> > > > List<List<ClusterNode>> where index of element in returned list
> > > corresponds
> > > > to partition number. Assignment for each partition represented as
> list of
> > > > nodes where primary node is always the first. So you can use existing
> > > > affinity functions for you case just passing properly formed
> > > > AffinityFunctionContext to assignPartitions method.
> > > >
> > > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > The affinity function only maps a key to a partition id and it
> doesn't
> > > > > seem to provide a way to map the partition id to a cluster node. So
> > > I'm a
> > > > > little bit confused right now.
> > > > >
> > > > > Could you please clarify?
> > > > >
> > > > > Thanks a lot
> > > > > Andrey
> > > > >
> > > > > > From: [hidden email]
> > > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > > Subject: Re: Cluster group affinity
> > > > > > To: [hidden email]
> > > > > >
> > > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I have a user-defined cluster group and I'd like to be able to
> > > > > > > consistently pick the same node in the group for a given key.
> > > > > Essentially,
> > > > > > > what I want is a cluster group affinity that is not associated
> > > with any
> > > > > > > cache. How can I do it?
> > > > > > >
> > > > > >
> > > > > > Andrey, perhaps you could just take our affinity function and
> use it
> > > > > > directly, no?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Andrey
> > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Andrey Gura
> > > > GridGain Systems, Inc.
> > > > www.gridgain.com
> > >
> > >
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
>
>

--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Andrey Kornev

RE: Cluster group affinity

In reply to this post by dsetrakyan

Dmitriy,

This approach would definitely work, if it wasn't for the fact that the cluster groups in my case are created dynamically and may include any combination of nodes in the cluster (where the number of combinations grows exponentially with the number of nodes in the cluster). I don't think it's practical to create that many caches.

I still can't get why the affinity function can't be applied to an arbitrary cluster group, and why it must necessarily be a cache. Isn't the cache affinity just a special case of the cluster group affinity defined as ClusterGroup.forCache()?

Thanks
Andrey

> From: [hidden email]
> Date: Tue, 6 Oct 2015 12:07:39 -0700
> Subject: Re: Cluster group affinity
> To: [hidden email]
>
> On Tue, Oct 6, 2015 at 8:46 AM, Andrey Kornev <[hidden email]>
> wrote:
>
> > Thanks, Andrey! This definitely helps.
> >
> > It's just that implementing such a simple feature in the "user space"
> > feels awkward and requires intimate knowledge of fairly low-level details
> > of how things work in the current version.
> >
> > Just curios, how about providing an override for Ignite.affinity() method
> > that ClusterGroup? Is there something fundamentally wrong about calculating
> > the affinity for an arbitrary collection of nodes (such as a ClusterGroup
> > is)?
> >
>
> Affinity is usually associated with data. In your case you have no data,
> but you still need keys to be always mapped to the same node. How about
> creating an empty cache and using standard cache API for determining the
> affinity for a key?
>
>
> > Regards
> > Andrey
> >
> > > Date: Tue, 6 Oct 2015 18:12:48 +0300
> > > Subject: Re: Cluster group affinity
> > > From: [hidden email]
> > > To: [hidden email]
> > >
> > > Andrey,
> > >
> > >
> > > > 1) I'm expected to return an instance of the internal class
> > > > AffinityTopologyVersion.
> > >
> > >
> > > If you are talking about AffinityContextFunction.currentTopologyVersion
> > > method then for now this method is nowhere uses. But it make sense to
> > > return non null value in order to avoid problems in the future.
> > >
> > > 2) the consequences of returning null from
> > > > AffinityFunctionContext.previousAssignment and
> > > > AffinityFunctionContext.discoveryEvent methods (because I can't
> > provide any
> > > > meaningful implementation for them) are not clear.
> > > >
> > >
> > > Both methods declared as @Nullable, so affinity function developer should
> > > correctly handle this cases. In Ignite only FairAffinityFunction uses
> > these
> > > methods. FairAffinityFunction tries to obtain left node Id from event of
> > > EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs to
> > > exclude this node assignment from previous assignments. So if your
> > cluster
> > > group lost node you can return EVT_NODE_LEFT discovery event with Id of
> > > lost node from discoveryEvent method and assignments for previous cluster
> > > group state from previousAssignment method.
> > >
> > > RendezvousAffinityFunction uses only currentTopologySnapshot() and
> > > backups() methods of AffinityFunctionContext interface.
> > >
> > >
> > > On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <[hidden email]>
> > > wrote:
> > >
> > > > Andrey, thanks!
> > > >
> > > > But a "properly formed AffinityFunctionContext" is the problem:
> > > > 1) I'm expected to return an instance of the internal class
> > > > AffinityTopologyVersion.
> > > > 2) the consequences of returning null from
> > > > AffinityFunctionContext.previousAssignment and
> > > > AffinityFunctionContext.discoveryEvent methods (because I can't
> > provide any
> > > > meaningful implementation for them) are not clear.
> > > >
> > > > Please advise.
> > > >
> > > > Thanks
> > > > Andrey
> > > >
> > > > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > > > Subject: Re: Cluster group affinity
> > > > > From: [hidden email]
> > > > > To: [hidden email]
> > > > >
> > > > > Andrey,
> > > > >
> > > > > See AffinityFunction.assignPartitions method. It returns assignment
> > list
> > > > as
> > > > > List<List<ClusterNode>> where index of element in returned list
> > > > corresponds
> > > > > to partition number. Assignment for each partition represented as
> > list of
> > > > > nodes where primary node is always the first. So you can use existing
> > > > > affinity functions for you case just passing properly formed
> > > > > AffinityFunctionContext to assignPartitions method.
> > > > >
> > > > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > The affinity function only maps a key to a partition id and it
> > doesn't
> > > > > > seem to provide a way to map the partition id to a cluster node. So
> > > > I'm a
> > > > > > little bit confused right now.
> > > > > >
> > > > > > Could you please clarify?
> > > > > >
> > > > > > Thanks a lot
> > > > > > Andrey
> > > > > >
> > > > > > > From: [hidden email]
> > > > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > > > Subject: Re: Cluster group affinity
> > > > > > > To: [hidden email]
> > > > > > >
> > > > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I have a user-defined cluster group and I'd like to be able to
> > > > > > > > consistently pick the same node in the group for a given key.
> > > > > > Essentially,
> > > > > > > > what I want is a cluster group affinity that is not associated
> > > > with any
> > > > > > > > cache. How can I do it?
> > > > > > > >
> > > > > > >
> > > > > > > Andrey, perhaps you could just take our affinity function and
> > use it
> > > > > > > directly, no?
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Andrey
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Andrey Gura
> > > > > GridGain Systems, Inc.
> > > > > www.gridgain.com
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Andrey Gura
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> >
> >

yzhdanov

Re: Cluster group affinity

Andrey, probably it is, but I am not sure if I have ever had a thought for
mentioned scenario.

I think you should get existing implementation and use it like this. Make
sure to cache assignments as this may be quiet expensive operation.

Ignite ignite = Ignition.start(cfg);

ClusterGroup group = ignite.cluster().forPredicate(new Predicate());

final List<ClusterNode> snapshot = new ArrayList<>(group.nodes());

RendezvousAffinityFunction aff = new RendezvousAffinityFunction();

List<List<ClusterNode>> parts = aff.assignPartitions(new
AffinityFunctionContext() {
@Nullable @Override public List<ClusterNode>
previousAssignment(int part) {
return null;
}

@Override public int backups() {
return 0;
}

@Override public List<ClusterNode> currentTopologySnapshot() {
return snapshot;
}

@Override public AffinityTopologyVersion
currentTopologyVersion() {
return null;
}

@Nullable @Override public DiscoveryEvent discoveryEvent() {
return null;
}
});

// Picking node.
ClusterNode node = parts.get(aff.partition(key)).get(0);

--Yakov

2015-10-07 11:39 GMT+03:00 Andrey Kornev <[hidden email]>:

> Dmitriy,
>
> This approach would definitely work, if it wasn't for the fact that the
> cluster groups in my case are created dynamically and may include any
> combination of nodes in the cluster (where the number of combinations grows
> exponentially with the number of nodes in the cluster). I don't think it's
> practical to create that many caches.
>
> I still can't get why the affinity function can't be applied to an
> arbitrary cluster group, and why it must necessarily be a cache. Isn't the
> cache affinity just a special case of the cluster group affinity defined as
> ClusterGroup.forCache()?
>
> Thanks
> Andrey
>
> > From: [hidden email]
> > Date: Tue, 6 Oct 2015 12:07:39 -0700
> > Subject: Re: Cluster group affinity
> > To: [hidden email]
> >
> > On Tue, Oct 6, 2015 at 8:46 AM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Thanks, Andrey! This definitely helps.
> > >
> > > It's just that implementing such a simple feature in the "user space"
> > > feels awkward and requires intimate knowledge of fairly low-level
> details
> > > of how things work in the current version.
> > >
> > > Just curios, how about providing an override for Ignite.affinity()
> method
> > > that ClusterGroup? Is there something fundamentally wrong about
> calculating
> > > the affinity for an arbitrary collection of nodes (such as a
> ClusterGroup
> > > is)?
> > >
> >
> > Affinity is usually associated with data. In your case you have no data,
> > but you still need keys to be always mapped to the same node. How about
> > creating an empty cache and using standard cache API for determining the
> > affinity for a key?
> >
> >
> > > Regards
> > > Andrey
> > >
> > > > Date: Tue, 6 Oct 2015 18:12:48 +0300
> > > > Subject: Re: Cluster group affinity
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Andrey,
> > > >
> > > >
> > > > > 1) I'm expected to return an instance of the internal class
> > > > > AffinityTopologyVersion.
> > > >
> > > >
> > > > If you are talking about
> AffinityContextFunction.currentTopologyVersion
> > > > method then for now this method is nowhere uses. But it make sense to
> > > > return non null value in order to avoid problems in the future.
> > > >
> > > > 2) the consequences of returning null from
> > > > > AffinityFunctionContext.previousAssignment and
> > > > > AffinityFunctionContext.discoveryEvent methods (because I can't
> > > provide any
> > > > > meaningful implementation for them) are not clear.
> > > > >
> > > >
> > > > Both methods declared as @Nullable, so affinity function developer
> should
> > > > correctly handle this cases. In Ignite only FairAffinityFunction uses
> > > these
> > > > methods. FairAffinityFunction tries to obtain left node Id from
> event of
> > > > EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs
> to
> > > > exclude this node assignment from previous assignments. So if your
> > > cluster
> > > > group lost node you can return EVT_NODE_LEFT discovery event with Id
> of
> > > > lost node from discoveryEvent method and assignments for previous
> cluster
> > > > group state from previousAssignment method.
> > > >
> > > > RendezvousAffinityFunction uses only currentTopologySnapshot() and
> > > > backups() methods of AffinityFunctionContext interface.
> > > >
> > > >
> > > > On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Andrey, thanks!
> > > > >
> > > > > But a "properly formed AffinityFunctionContext" is the problem:
> > > > > 1) I'm expected to return an instance of the internal class
> > > > > AffinityTopologyVersion.
> > > > > 2) the consequences of returning null from
> > > > > AffinityFunctionContext.previousAssignment and
> > > > > AffinityFunctionContext.discoveryEvent methods (because I can't
> > > provide any
> > > > > meaningful implementation for them) are not clear.
> > > > >
> > > > > Please advise.
> > > > >
> > > > > Thanks
> > > > > Andrey
> > > > >
> > > > > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > > > > Subject: Re: Cluster group affinity
> > > > > > From: [hidden email]
> > > > > > To: [hidden email]
> > > > > >
> > > > > > Andrey,
> > > > > >
> > > > > > See AffinityFunction.assignPartitions method. It returns
> assignment
> > > list
> > > > > as
> > > > > > List<List<ClusterNode>> where index of element in returned list
> > > > > corresponds
> > > > > > to partition number. Assignment for each partition represented as
> > > list of
> > > > > > nodes where primary node is always the first. So you can use
> existing
> > > > > > affinity functions for you case just passing properly formed
> > > > > > AffinityFunctionContext to assignPartitions method.
> > > > > >
> > > > > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > The affinity function only maps a key to a partition id and it
> > > doesn't
> > > > > > > seem to provide a way to map the partition id to a cluster
> node. So
> > > > > I'm a
> > > > > > > little bit confused right now.
> > > > > > >
> > > > > > > Could you please clarify?
> > > > > > >
> > > > > > > Thanks a lot
> > > > > > > Andrey
> > > > > > >
> > > > > > > > From: [hidden email]
> > > > > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > > > > Subject: Re: Cluster group affinity
> > > > > > > > To: [hidden email]
> > > > > > > >
> > > > > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > I have a user-defined cluster group and I'd like to be
> able to
> > > > > > > > > consistently pick the same node in the group for a given
> key.
> > > > > > > Essentially,
> > > > > > > > > what I want is a cluster group affinity that is not
> associated
> > > > > with any
> > > > > > > > > cache. How can I do it?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Andrey, perhaps you could just take our affinity function and
> > > use it
> > > > > > > > directly, no?
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Andrey Gura
> > > > > > GridGain Systems, Inc.
> > > > > > www.gridgain.com
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Andrey Gura
> > > > GridGain Systems, Inc.
> > > > www.gridgain.com
> > >
> > >
>
>

Andrey Kornev

RE: Cluster group affinity

Thanks, Yakov! I'm going to use your code (I hope it's Apache2 licensed:)))

As for the use case, it's quite simple. I have a group of compute nodes that are being sent a stream of work items and I'd like to parallelize processing by partitioning the stream across all the nodes in the group by some attribute. This is pure compute, no caches.

Thanks
Andrey

> Date: Wed, 7 Oct 2015 13:46:23 +0300
> Subject: Re: Cluster group affinity
> From: [hidden email]
> To: [hidden email]
>
> Andrey, probably it is, but I am not sure if I have ever had a thought for
> mentioned scenario.
>
> I think you should get existing implementation and use it like this. Make
> sure to cache assignments as this may be quiet expensive operation.
>
> Ignite ignite = Ignition.start(cfg);
>
> ClusterGroup group = ignite.cluster().forPredicate(new Predicate());
>
> final List<ClusterNode> snapshot = new ArrayList<>(group.nodes());
>
> RendezvousAffinityFunction aff = new RendezvousAffinityFunction();
>
> List<List<ClusterNode>> parts = aff.assignPartitions(new
> AffinityFunctionContext() {
> @Nullable @Override public List<ClusterNode>
> previousAssignment(int part) {
> return null;
> }
>
> @Override public int backups() {
> return 0;
> }
>
> @Override public List<ClusterNode> currentTopologySnapshot() {
> return snapshot;
> }
>
> @Override public AffinityTopologyVersion
> currentTopologyVersion() {
> return null;
> }
>
> @Nullable @Override public DiscoveryEvent discoveryEvent() {
> return null;
> }
> });
>
> // Picking node.
> ClusterNode node = parts.get(aff.partition(key)).get(0);
>
>
> --Yakov
>
> 2015-10-07 11:39 GMT+03:00 Andrey Kornev <[hidden email]>:
>
> > Dmitriy,
> >
> > This approach would definitely work, if it wasn't for the fact that the
> > cluster groups in my case are created dynamically and may include any
> > combination of nodes in the cluster (where the number of combinations grows
> > exponentially with the number of nodes in the cluster). I don't think it's
> > practical to create that many caches.
> >
> > I still can't get why the affinity function can't be applied to an
> > arbitrary cluster group, and why it must necessarily be a cache. Isn't the
> > cache affinity just a special case of the cluster group affinity defined as
> > ClusterGroup.forCache()?
> >
> > Thanks
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Tue, 6 Oct 2015 12:07:39 -0700
> > > Subject: Re: Cluster group affinity
> > > To: [hidden email]
> > >
> > > On Tue, Oct 6, 2015 at 8:46 AM, Andrey Kornev <[hidden email]>
> > > wrote:
> > >
> > > > Thanks, Andrey! This definitely helps.
> > > >
> > > > It's just that implementing such a simple feature in the "user space"
> > > > feels awkward and requires intimate knowledge of fairly low-level
> > details
> > > > of how things work in the current version.
> > > >
> > > > Just curios, how about providing an override for Ignite.affinity()
> > method
> > > > that ClusterGroup? Is there something fundamentally wrong about
> > calculating
> > > > the affinity for an arbitrary collection of nodes (such as a
> > ClusterGroup
> > > > is)?
> > > >
> > >
> > > Affinity is usually associated with data. In your case you have no data,
> > > but you still need keys to be always mapped to the same node. How about
> > > creating an empty cache and using standard cache API for determining the
> > > affinity for a key?
> > >
> > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > Date: Tue, 6 Oct 2015 18:12:48 +0300
> > > > > Subject: Re: Cluster group affinity
> > > > > From: [hidden email]
> > > > > To: [hidden email]
> > > > >
> > > > > Andrey,
> > > > >
> > > > >
> > > > > > 1) I'm expected to return an instance of the internal class
> > > > > > AffinityTopologyVersion.
> > > > >
> > > > >
> > > > > If you are talking about
> > AffinityContextFunction.currentTopologyVersion
> > > > > method then for now this method is nowhere uses. But it make sense to
> > > > > return non null value in order to avoid problems in the future.
> > > > >
> > > > > 2) the consequences of returning null from
> > > > > > AffinityFunctionContext.previousAssignment and
> > > > > > AffinityFunctionContext.discoveryEvent methods (because I can't
> > > > provide any
> > > > > > meaningful implementation for them) are not clear.
> > > > > >
> > > > >
> > > > > Both methods declared as @Nullable, so affinity function developer
> > should
> > > > > correctly handle this cases. In Ignite only FairAffinityFunction uses
> > > > these
> > > > > methods. FairAffinityFunction tries to obtain left node Id from
> > event of
> > > > > EventType.EVT_NODE_LEFT or EventType.EVT_NODE_FAILED type. It needs
> > to
> > > > > exclude this node assignment from previous assignments. So if your
> > > > cluster
> > > > > group lost node you can return EVT_NODE_LEFT discovery event with Id
> > of
> > > > > lost node from discoveryEvent method and assignments for previous
> > cluster
> > > > > group state from previousAssignment method.
> > > > >
> > > > > RendezvousAffinityFunction uses only currentTopologySnapshot() and
> > > > > backups() methods of AffinityFunctionContext interface.
> > > > >
> > > > >
> > > > > On Tue, Oct 6, 2015 at 5:07 PM, Andrey Kornev <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Andrey, thanks!
> > > > > >
> > > > > > But a "properly formed AffinityFunctionContext" is the problem:
> > > > > > 1) I'm expected to return an instance of the internal class
> > > > > > AffinityTopologyVersion.
> > > > > > 2) the consequences of returning null from
> > > > > > AffinityFunctionContext.previousAssignment and
> > > > > > AffinityFunctionContext.discoveryEvent methods (because I can't
> > > > provide any
> > > > > > meaningful implementation for them) are not clear.
> > > > > >
> > > > > > Please advise.
> > > > > >
> > > > > > Thanks
> > > > > > Andrey
> > > > > >
> > > > > > > Date: Tue, 6 Oct 2015 16:43:10 +0300
> > > > > > > Subject: Re: Cluster group affinity
> > > > > > > From: [hidden email]
> > > > > > > To: [hidden email]
> > > > > > >
> > > > > > > Andrey,
> > > > > > >
> > > > > > > See AffinityFunction.assignPartitions method. It returns
> > assignment
> > > > list
> > > > > > as
> > > > > > > List<List<ClusterNode>> where index of element in returned list
> > > > > > corresponds
> > > > > > > to partition number. Assignment for each partition represented as
> > > > list of
> > > > > > > nodes where primary node is always the first. So you can use
> > existing
> > > > > > > affinity functions for you case just passing properly formed
> > > > > > > AffinityFunctionContext to assignPartitions method.
> > > > > > >
> > > > > > > On Tue, Oct 6, 2015 at 4:25 PM, Andrey Kornev <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Dmitriy,
> > > > > > > >
> > > > > > > > The affinity function only maps a key to a partition id and it
> > > > doesn't
> > > > > > > > seem to provide a way to map the partition id to a cluster
> > node. So
> > > > > > I'm a
> > > > > > > > little bit confused right now.
> > > > > > > >
> > > > > > > > Could you please clarify?
> > > > > > > >
> > > > > > > > Thanks a lot
> > > > > > > > Andrey
> > > > > > > >
> > > > > > > > > From: [hidden email]
> > > > > > > > > Date: Mon, 5 Oct 2015 09:53:25 -0700
> > > > > > > > > Subject: Re: Cluster group affinity
> > > > > > > > > To: [hidden email]
> > > > > > > > >
> > > > > > > > > On Mon, Oct 5, 2015 at 2:28 AM, Andrey Kornev <
> > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > I have a user-defined cluster group and I'd like to be
> > able to
> > > > > > > > > > consistently pick the same node in the group for a given
> > key.
> > > > > > > > Essentially,
> > > > > > > > > > what I want is a cluster group affinity that is not
> > associated
> > > > > > with any
> > > > > > > > > > cache. How can I do it?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Andrey, perhaps you could just take our affinity function and
> > > > use it
> > > > > > > > > directly, no?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Andrey
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Andrey Gura
> > > > > > > GridGain Systems, Inc.
> > > > > > > www.gridgain.com
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Andrey Gura
> > > > > GridGain Systems, Inc.
> > > > > www.gridgain.com
> > > >
> > > >
> >
> >

yzhdanov

Re: Cluster group affinity

2015-10-07 15:14 GMT+03:00 Andrey Kornev <[hidden email]>:

> Thanks, Yakov! I'm going to use your code (I hope it's Apache2 licensed:)))
>

You are welcome! No warranties, btw. I did not test this code :)

>
> As for the use case, it's quite simple. I have a group of compute nodes
> that are being sent a stream of work items and I'd like to parallelize
> processing by partitioning the stream across all the nodes in the group by
> some attribute. This is pure compute, no caches.
>
> Thanks
> Andrey
>

--Yakov