Thin Client Kubernetes discovery

[Maksim Timonin]

Hi, Igniters

I'm now discovering the issue
https://issues.apache.org/jira/browse/IGNITE-13204 and wondering what is a
case that requires a client to be able to discover a cluster?

I believe that discovery is a goal of kubernetes itself. We could assign a
DNS name for a Service [1] and then any request to the Service will be
forwarded to any of Ignite pods behind it. So there is no need to extract
pods IP to provide the connection. For example, I've configured a  k8s
service with name "ignite" and port mapping 10800:9888. Then just connect
to it with sqlline by a string "ignite:9888".

I think that it's OK that clients should be configured with those settings
as they aren't changed during cluster live.

Also k8s provides SessionAffinity [2] to guarantee forwarding traffic from
one client to the same pod for every request. So there is no need to
provide IP for a specific pod if a state is required.

Maybe I'm missing something about this feature? Could somebody provide more
details for this task?

Below is the example config and connection string:

apiVersion: v1
kind: Service
metadata:
  name: ignite
spec:
  ports:
    - port: 9888
      targetPort: 10800
  selector:
    app: ignite

./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
0: jdbc:ignite:thin://ignite:9888>

[1] https://kubernetes.io/docs/concepts/services-networking/service/
[2]
https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace

[dmagda]

Max,

That improvement automates the cluster discovery if both the cluster and
thin clients are deployed *inside* of Kubernetes. That's my reading of the
ticket's description. While you're referring to the case when the client is
deployed outside of a K8S environment.

@Vladimir Pligin <[hidden email]>, could you please join the thread.
I've seen you joined the discussion in JIRA.

-
Denis


On Wed, Aug 12, 2020 at 12:06 PM Max Timonin <[hidden email]>
wrote:

> Hi, Igniters
>
> I'm now discovering the issue
> https://issues.apache.org/jira/browse/IGNITE-13204 and wondering what is a
> case that requires a client to be able to discover a cluster?
>
> I believe that discovery is a goal of kubernetes itself. We could assign a
> DNS name for a Service [1] and then any request to the Service will be
> forwarded to any of Ignite pods behind it. So there is no need to extract
> pods IP to provide the connection. For example, I've configured a  k8s
> service with name "ignite" and port mapping 10800:9888. Then just connect
> to it with sqlline by a string "ignite:9888".
>
> I think that it's OK that clients should be configured with those settings
> as they aren't changed during cluster live.
>
> Also k8s provides SessionAffinity [2] to guarantee forwarding traffic from
> one client to the same pod for every request. So there is no need to
> provide IP for a specific pod if a state is required.
>
> Maybe I'm missing something about this feature? Could somebody provide more
> details for this task?
>
> Below is the example config and connection string:
>
> apiVersion: v1
> kind: Service
> metadata:
>   name: ignite
> spec:
>   ports:
>     - port: 9888
>       targetPort: 10800
>   selector:
>     app: ignite
>
> ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> 0: jdbc:ignite:thin://ignite:9888>
>
> [1] https://kubernetes.io/docs/concepts/services-networking/service/
> [2]
>
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
>

[Maksim Timonin]

I'm not sure what you mean by inside/outside of kubernetes. Service name is
visible within k8s environment. I've described a case with connection from
another pod that is part of k8s cluster.

To provide connection outside of Kubernetes one should configure Ingress
[1]. It will have a fixed address assigned by a cluster administrator. Any
request outside of k8s to the Ingress will be proxied inside k8s env to the
Service and then to a pod automatically without need to provide any IP
addresses in a client configuration.

[1] https://kubernetes.io/docs/concepts/services-networking/ingress/

On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <[hidden email]> wrote:

> Max,
>
> That improvement automates the cluster discovery if both the cluster and
> thin clients are deployed *inside* of Kubernetes. That's my reading of the
> ticket's description. While you're referring to the case when the client is
> deployed outside of a K8S environment.
>
> @Vladimir Pligin <[hidden email]>, could you please join the thread.
> I've seen you joined the discussion in JIRA.
>
> -
> Denis
>
>
> On Wed, Aug 12, 2020 at 12:06 PM Max Timonin <[hidden email]>
> wrote:
>
> > Hi, Igniters
> >
> > I'm now discovering the issue
> > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering what
> is a
> > case that requires a client to be able to discover a cluster?
> >
> > I believe that discovery is a goal of kubernetes itself. We could assign
> a
> > DNS name for a Service [1] and then any request to the Service will be
> > forwarded to any of Ignite pods behind it. So there is no need to extract
> > pods IP to provide the connection. For example, I've configured a  k8s
> > service with name "ignite" and port mapping 10800:9888. Then just connect
> > to it with sqlline by a string "ignite:9888".
> >
> > I think that it's OK that clients should be configured with those
> settings
> > as they aren't changed during cluster live.
> >
> > Also k8s provides SessionAffinity [2] to guarantee forwarding traffic
> from
> > one client to the same pod for every request. So there is no need to
> > provide IP for a specific pod if a state is required.
> >
> > Maybe I'm missing something about this feature? Could somebody provide
> more
> > details for this task?
> >
> > Below is the example config and connection string:
> >
> > apiVersion: v1
> > kind: Service
> > metadata:
> >   name: ignite
> > spec:
> >   ports:
> >     - port: 9888
> >       targetPort: 10800
> >   selector:
> >     app: ignite
> >
> > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> > 0: jdbc:ignite:thin://ignite:9888>
> >
> > [1] https://kubernetes.io/docs/concepts/services-networking/service/
> > [2]
> >
> >
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
> >
>

[dmagda]

We're discussing the case when the thin client and server nodes are a part
of a single Kubernetes cluster.

Ignite thick client uses KubernetesIPFinder
<https://apacheignite.readme.io/docs/kubernetes-ip-finder> to auto-discover
other Ignite pods. The IP finder gets IP addresses of other Ignite pods via
a special Service instance, and then the thick client uses those addresses
to communicate to the rest of the cluster bypassing the Service. I
believe the ultimate goal of the JIRA task is to support a similar
capability for the thin client.

However, you solution might be the way to go. Still, I wonder how the thin
client will handle cases when a server pod, the client was connected to,
goes down, and the Service connects the client to another pod (considering
the session affinity option). It might break some internal state of the
connection on the client-side and lead to unpredictable behavior. Also, if
the partition-awareness feature is enabled, then the thin client will start
bypassing the Service for most of the key-value requests connecting to the
nodes directly using their IP addresses.

-
Denis


On Wed, Aug 12, 2020 at 11:07 PM Max Timonin <[hidden email]>
wrote:

> I'm not sure what you mean by inside/outside of kubernetes. Service name is
> visible within k8s environment. I've described a case with connection from
> another pod that is part of k8s cluster.
>
> To provide connection outside of Kubernetes one should configure Ingress
> [1]. It will have a fixed address assigned by a cluster administrator. Any
> request outside of k8s to the Ingress will be proxied inside k8s env to the
> Service and then to a pod automatically without need to provide any IP
> addresses in a client configuration.
>
> [1] https://kubernetes.io/docs/concepts/services-networking/ingress/
>
> On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <[hidden email]> wrote:
>
> > Max,
> >
> > That improvement automates the cluster discovery if both the cluster and
> > thin clients are deployed *inside* of Kubernetes. That's my reading of
> the
> > ticket's description. While you're referring to the case when the client
> is
> > deployed outside of a K8S environment.
> >
> > @Vladimir Pligin <[hidden email]>, could you please join the
> thread.
> > I've seen you joined the discussion in JIRA.
> >
> > -
> > Denis
> >
> >
> > On Wed, Aug 12, 2020 at 12:06 PM Max Timonin <[hidden email]>
> > wrote:
> >
> > > Hi, Igniters
> > >
> > > I'm now discovering the issue
> > > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering what
> > is a
> > > case that requires a client to be able to discover a cluster?
> > >
> > > I believe that discovery is a goal of kubernetes itself. We could
> assign
> > a
> > > DNS name for a Service [1] and then any request to the Service will be
> > > forwarded to any of Ignite pods behind it. So there is no need to
> extract
> > > pods IP to provide the connection. For example, I've configured a  k8s
> > > service with name "ignite" and port mapping 10800:9888. Then just
> connect
> > > to it with sqlline by a string "ignite:9888".
> > >
> > > I think that it's OK that clients should be configured with those
> > settings
> > > as they aren't changed during cluster live.
> > >
> > > Also k8s provides SessionAffinity [2] to guarantee forwarding traffic
> > from
> > > one client to the same pod for every request. So there is no need to
> > > provide IP for a specific pod if a state is required.
> > >
> > > Maybe I'm missing something about this feature? Could somebody provide
> > more
> > > details for this task?
> > >
> > > Below is the example config and connection string:
> > >
> > > apiVersion: v1
> > > kind: Service
> > > metadata:
> > >   name: ignite
> > > spec:
> > >   ports:
> > >     - port: 9888
> > >       targetPort: 10800
> > >   selector:
> > >     app: ignite
> > >
> > > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> > > 0: jdbc:ignite:thin://ignite:9888>
> > >
> > > [1] https://kubernetes.io/docs/concepts/services-networking/service/
> > > [2]
> > >
> > >
> >
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
> > >
> >
>

[Pavel Tupitsyn]

Max, Denis,

Partition Awareness [1] is our main reason to have a specialized k8s
discovery mechanism in Thin Clients.

k8s ingress mechanism is fine for a single client connection, but in
Partition Aware mode
thin clients need to connect to every server node, and track nodes as they
enter or leave the topology.

Denis is right, current tickets [2] [3] are about Thin Clients being
deployed in the same k8s cluster as server nodes.
Using k8s APIs we can get every pod address and establish connections.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
[2] https://issues.apache.org/jira/browse/IGNITE-13011
<https://issues.apache.org/jira/browse/IGNITE-13204>
[3] https://issues.apache.org/jira/browse/IGNITE-13204

On Thu, Aug 13, 2020 at 10:48 AM Denis Magda <[hidden email]> wrote:

> We're discussing the case when the thin client and server nodes are a part
> of a single Kubernetes cluster.
>
> Ignite thick client uses KubernetesIPFinder
> <https://apacheignite.readme.io/docs/kubernetes-ip-finder> to
> auto-discover
> other Ignite pods. The IP finder gets IP addresses of other Ignite pods via
> a special Service instance, and then the thick client uses those addresses
> to communicate to the rest of the cluster bypassing the Service. I
> believe the ultimate goal of the JIRA task is to support a similar
> capability for the thin client.
>
> However, you solution might be the way to go. Still, I wonder how the thin
> client will handle cases when a server pod, the client was connected to,
> goes down, and the Service connects the client to another pod (considering
> the session affinity option). It might break some internal state of the
> connection on the client-side and lead to unpredictable behavior. Also, if
> the partition-awareness feature is enabled, then the thin client will start
> bypassing the Service for most of the key-value requests connecting to the
> nodes directly using their IP addresses.
>
> -
> Denis
>
>
> On Wed, Aug 12, 2020 at 11:07 PM Max Timonin <[hidden email]>
> wrote:
>
> > I'm not sure what you mean by inside/outside of kubernetes. Service name
> is
> > visible within k8s environment. I've described a case with connection
> from
> > another pod that is part of k8s cluster.
> >
> > To provide connection outside of Kubernetes one should configure Ingress
> > [1]. It will have a fixed address assigned by a cluster administrator.
> Any
> > request outside of k8s to the Ingress will be proxied inside k8s env to
> the
> > Service and then to a pod automatically without need to provide any IP
> > addresses in a client configuration.
> >
> > [1] https://kubernetes.io/docs/concepts/services-networking/ingress/
> >
> > On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <[hidden email]> wrote:
> >
> > > Max,
> > >
> > > That improvement automates the cluster discovery if both the cluster
> and
> > > thin clients are deployed *inside* of Kubernetes. That's my reading of
> > the
> > > ticket's description. While you're referring to the case when the
> client
> > is
> > > deployed outside of a K8S environment.
> > >
> > > @Vladimir Pligin <[hidden email]>, could you please join the
> > thread.
> > > I've seen you joined the discussion in JIRA.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Wed, Aug 12, 2020 at 12:06 PM Max Timonin <[hidden email]>
> > > wrote:
> > >
> > > > Hi, Igniters
> > > >
> > > > I'm now discovering the issue
> > > > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering
> what
> > > is a
> > > > case that requires a client to be able to discover a cluster?
> > > >
> > > > I believe that discovery is a goal of kubernetes itself. We could
> > assign
> > > a
> > > > DNS name for a Service [1] and then any request to the Service will
> be
> > > > forwarded to any of Ignite pods behind it. So there is no need to
> > extract
> > > > pods IP to provide the connection. For example, I've configured a
> k8s
> > > > service with name "ignite" and port mapping 10800:9888. Then just
> > connect
> > > > to it with sqlline by a string "ignite:9888".
> > > >
> > > > I think that it's OK that clients should be configured with those
> > > settings
> > > > as they aren't changed during cluster live.
> > > >
> > > > Also k8s provides SessionAffinity [2] to guarantee forwarding traffic
> > > from
> > > > one client to the same pod for every request. So there is no need to
> > > > provide IP for a specific pod if a state is required.
> > > >
> > > > Maybe I'm missing something about this feature? Could somebody
> provide
> > > more
> > > > details for this task?
> > > >
> > > > Below is the example config and connection string:
> > > >
> > > > apiVersion: v1
> > > > kind: Service
> > > > metadata:
> > > >   name: ignite
> > > > spec:
> > > >   ports:
> > > >     - port: 9888
> > > >       targetPort: 10800
> > > >   selector:
> > > >     app: ignite
> > > >
> > > > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> > > > 0: jdbc:ignite:thin://ignite:9888>
> > > >
> > > > [1] https://kubernetes.io/docs/concepts/services-networking/service/
> > > > [2]
> > > >
> > > >
> > >
> >
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
> > > >
> > >
> >
>

[Maksim Timonin]

In case of an enabled partition-awareness feature or sessions I suppose
that a state appears. And as we have a state we should use the stateful
deployment of kubernetes with StatefulSet [1]. In that case addresses of
nodes are predictable and aren't changed so one can configure a client with
them. I think it's a better solution than implementing specific discovery
by self. Point is to rely on k8s discovery mechanisms as much as possible
otherwise why use it? What do you think?

I will prepare a demo with an enabled partition-awareness feature and
configured StatefulSet. If you see any issues with it please let me know.

[1] https://apacheignite.readme.io/docs/stateful-deployment

On Thu, Aug 13, 2020 at 11:25 AM Pavel Tupitsyn <[hidden email]>
wrote:

> Max, Denis,
>
> Partition Awareness [1] is our main reason to have a specialized k8s
> discovery mechanism in Thin Clients.
>
> k8s ingress mechanism is fine for a single client connection, but in
> Partition Aware mode
> thin clients need to connect to every server node, and track nodes as they
> enter or leave the topology.
>
> Denis is right, current tickets [2] [3] are about Thin Clients being
> deployed in the same k8s cluster as server nodes.
> Using k8s APIs we can get every pod address and establish connections.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> [2] https://issues.apache.org/jira/browse/IGNITE-13011
> <https://issues.apache.org/jira/browse/IGNITE-13204>
> [3] https://issues.apache.org/jira/browse/IGNITE-13204
>
> On Thu, Aug 13, 2020 at 10:48 AM Denis Magda <[hidden email]> wrote:
>
> > We're discussing the case when the thin client and server nodes are a
> part
> > of a single Kubernetes cluster.
> >
> > Ignite thick client uses KubernetesIPFinder
> > <https://apacheignite.readme.io/docs/kubernetes-ip-finder> to
> > auto-discover
> > other Ignite pods. The IP finder gets IP addresses of other Ignite pods
> via
> > a special Service instance, and then the thick client uses those
> addresses
> > to communicate to the rest of the cluster bypassing the Service. I
> > believe the ultimate goal of the JIRA task is to support a similar
> > capability for the thin client.
> >
> > However, you solution might be the way to go. Still, I wonder how the
> thin
> > client will handle cases when a server pod, the client was connected to,
> > goes down, and the Service connects the client to another pod
> (considering
> > the session affinity option). It might break some internal state of the
> > connection on the client-side and lead to unpredictable behavior. Also,
> if
> > the partition-awareness feature is enabled, then the thin client will
> start
> > bypassing the Service for most of the key-value requests connecting to
> the
> > nodes directly using their IP addresses.
> >
> > -
> > Denis
> >
> >
> > On Wed, Aug 12, 2020 at 11:07 PM Max Timonin <[hidden email]>
> > wrote:
> >
> > > I'm not sure what you mean by inside/outside of kubernetes. Service
> name
> > is
> > > visible within k8s environment. I've described a case with connection
> > from
> > > another pod that is part of k8s cluster.
> > >
> > > To provide connection outside of Kubernetes one should configure
> Ingress
> > > [1]. It will have a fixed address assigned by a cluster administrator.
> > Any
> > > request outside of k8s to the Ingress will be proxied inside k8s env to
> > the
> > > Service and then to a pod automatically without need to provide any IP
> > > addresses in a client configuration.
> > >
> > > [1] https://kubernetes.io/docs/concepts/services-networking/ingress/
> > >
> > > On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <[hidden email]> wrote:
> > >
> > > > Max,
> > > >
> > > > That improvement automates the cluster discovery if both the cluster
> > and
> > > > thin clients are deployed *inside* of Kubernetes. That's my reading
> of
> > > the
> > > > ticket's description. While you're referring to the case when the
> > client
> > > is
> > > > deployed outside of a K8S environment.
> > > >
> > > > @Vladimir Pligin <[hidden email]>, could you please join the
> > > thread.
> > > > I've seen you joined the discussion in JIRA.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Wed, Aug 12, 2020 at 12:06 PM Max Timonin <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi, Igniters
> > > > >
> > > > > I'm now discovering the issue
> > > > > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering
> > what
> > > > is a
> > > > > case that requires a client to be able to discover a cluster?
> > > > >
> > > > > I believe that discovery is a goal of kubernetes itself. We could
> > > assign
> > > > a
> > > > > DNS name for a Service [1] and then any request to the Service will
> > be
> > > > > forwarded to any of Ignite pods behind it. So there is no need to
> > > extract
> > > > > pods IP to provide the connection. For example, I've configured a
> > k8s
> > > > > service with name "ignite" and port mapping 10800:9888. Then just
> > > connect
> > > > > to it with sqlline by a string "ignite:9888".
> > > > >
> > > > > I think that it's OK that clients should be configured with those
> > > > settings
> > > > > as they aren't changed during cluster live.
> > > > >
> > > > > Also k8s provides SessionAffinity [2] to guarantee forwarding
> traffic
> > > > from
> > > > > one client to the same pod for every request. So there is no need
> to
> > > > > provide IP for a specific pod if a state is required.
> > > > >
> > > > > Maybe I'm missing something about this feature? Could somebody
> > provide
> > > > more
> > > > > details for this task?
> > > > >
> > > > > Below is the example config and connection string:
> > > > >
> > > > > apiVersion: v1
> > > > > kind: Service
> > > > > metadata:
> > > > >   name: ignite
> > > > > spec:
> > > > >   ports:
> > > > >     - port: 9888
> > > > >       targetPort: 10800
> > > > >   selector:
> > > > >     app: ignite
> > > > >
> > > > > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> > > > > 0: jdbc:ignite:thin://ignite:9888>
> > > > >
> > > > > [1]
> https://kubernetes.io/docs/concepts/services-networking/service/
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
> > > > >
> > > >
> > >
> >
>

[Pavel Tupitsyn]

Max,

The point of Thin Client k8s discovery feature is to make k8s deployment
simple -
similar to KubernetesIPFinder for thick clients and servers.

- I don't think that enabling partition awareness is related to a
StatefulSet in any way
- Not everyone is able to / wants to define a StatefulSet
- Even with a StatefulSet you'll have to manually provide a set of all
server nodes IPs
  in the thin client config, which is extra upfront work and extra
maintenance

With k8s discovery I can set a single property and the whole thing just
works,
servers join the cluster, thin clients connect to them.

This is especially important for beginners. Ignite should be easy to get
started with.

On Thu, Aug 13, 2020 at 11:35 AM Max Timonin <[hidden email]>
wrote:

> In case of an enabled partition-awareness feature or sessions I suppose
> that a state appears. And as we have a state we should use the stateful
> deployment of kubernetes with StatefulSet [1]. In that case addresses of
> nodes are predictable and aren't changed so one can configure a client with
> them. I think it's a better solution than implementing specific discovery
> by self. Point is to rely on k8s discovery mechanisms as much as possible
> otherwise why use it? What do you think?
>
> I will prepare a demo with an enabled partition-awareness feature and
> configured StatefulSet. If you see any issues with it please let me know.
>
> [1] https://apacheignite.readme.io/docs/stateful-deployment
>
> On Thu, Aug 13, 2020 at 11:25 AM Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Max, Denis,
> >
> > Partition Awareness [1] is our main reason to have a specialized k8s
> > discovery mechanism in Thin Clients.
> >
> > k8s ingress mechanism is fine for a single client connection, but in
> > Partition Aware mode
> > thin clients need to connect to every server node, and track nodes as
> they
> > enter or leave the topology.
> >
> > Denis is right, current tickets [2] [3] are about Thin Clients being
> > deployed in the same k8s cluster as server nodes.
> > Using k8s APIs we can get every pod address and establish connections.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > [2] https://issues.apache.org/jira/browse/IGNITE-13011
> > <https://issues.apache.org/jira/browse/IGNITE-13204>
> > [3] https://issues.apache.org/jira/browse/IGNITE-13204
> >
> > On Thu, Aug 13, 2020 at 10:48 AM Denis Magda <[hidden email]> wrote:
> >
> > > We're discussing the case when the thin client and server nodes are a
> > part
> > > of a single Kubernetes cluster.
> > >
> > > Ignite thick client uses KubernetesIPFinder
> > > <https://apacheignite.readme.io/docs/kubernetes-ip-finder> to
> > > auto-discover
> > > other Ignite pods. The IP finder gets IP addresses of other Ignite pods
> > via
> > > a special Service instance, and then the thick client uses those
> > addresses
> > > to communicate to the rest of the cluster bypassing the Service. I
> > > believe the ultimate goal of the JIRA task is to support a similar
> > > capability for the thin client.
> > >
> > > However, you solution might be the way to go. Still, I wonder how the
> > thin
> > > client will handle cases when a server pod, the client was connected
> to,
> > > goes down, and the Service connects the client to another pod
> > (considering
> > > the session affinity option). It might break some internal state of the
> > > connection on the client-side and lead to unpredictable behavior. Also,
> > if
> > > the partition-awareness feature is enabled, then the thin client will
> > start
> > > bypassing the Service for most of the key-value requests connecting to
> > the
> > > nodes directly using their IP addresses.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Wed, Aug 12, 2020 at 11:07 PM Max Timonin <[hidden email]>
> > > wrote:
> > >
> > > > I'm not sure what you mean by inside/outside of kubernetes. Service
> > name
> > > is
> > > > visible within k8s environment. I've described a case with connection
> > > from
> > > > another pod that is part of k8s cluster.
> > > >
> > > > To provide connection outside of Kubernetes one should configure
> > Ingress
> > > > [1]. It will have a fixed address assigned by a cluster
> administrator.
> > > Any
> > > > request outside of k8s to the Ingress will be proxied inside k8s env
> to
> > > the
> > > > Service and then to a pod automatically without need to provide any
> IP
> > > > addresses in a client configuration.
> > > >
> > > > [1] https://kubernetes.io/docs/concepts/services-networking/ingress/
> > > >
> > > > On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <[hidden email]>
> wrote:
> > > >
> > > > > Max,
> > > > >
> > > > > That improvement automates the cluster discovery if both the
> cluster
> > > and
> > > > > thin clients are deployed *inside* of Kubernetes. That's my reading
> > of
> > > > the
> > > > > ticket's description. While you're referring to the case when the
> > > client
> > > > is
> > > > > deployed outside of a K8S environment.
> > > > >
> > > > > @Vladimir Pligin <[hidden email]>, could you please join the
> > > > thread.
> > > > > I've seen you joined the discussion in JIRA.
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Wed, Aug 12, 2020 at 12:06 PM Max Timonin <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi, Igniters
> > > > > >
> > > > > > I'm now discovering the issue
> > > > > > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering
> > > what
> > > > > is a
> > > > > > case that requires a client to be able to discover a cluster?
> > > > > >
> > > > > > I believe that discovery is a goal of kubernetes itself. We could
> > > > assign
> > > > > a
> > > > > > DNS name for a Service [1] and then any request to the Service
> will
> > > be
> > > > > > forwarded to any of Ignite pods behind it. So there is no need to
> > > > extract
> > > > > > pods IP to provide the connection. For example, I've configured a
> > > k8s
> > > > > > service with name "ignite" and port mapping 10800:9888. Then just
> > > > connect
> > > > > > to it with sqlline by a string "ignite:9888".
> > > > > >
> > > > > > I think that it's OK that clients should be configured with those
> > > > > settings
> > > > > > as they aren't changed during cluster live.
> > > > > >
> > > > > > Also k8s provides SessionAffinity [2] to guarantee forwarding
> > traffic
> > > > > from
> > > > > > one client to the same pod for every request. So there is no need
> > to
> > > > > > provide IP for a specific pod if a state is required.
> > > > > >
> > > > > > Maybe I'm missing something about this feature? Could somebody
> > > provide
> > > > > more
> > > > > > details for this task?
> > > > > >
> > > > > > Below is the example config and connection string:
> > > > > >
> > > > > > apiVersion: v1
> > > > > > kind: Service
> > > > > > metadata:
> > > > > >   name: ignite
> > > > > > spec:
> > > > > >   ports:
> > > > > >     - port: 9888
> > > > > >       targetPort: 10800
> > > > > >   selector:
> > > > > >     app: ignite
> > > > > >
> > > > > > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> > > > > > 0: jdbc:ignite:thin://ignite:9888>
> > > > > >
> > > > > > [1]
> > https://kubernetes.io/docs/concepts/services-networking/service/
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Vladimir Pligin]

Hi guys,

Maybe I'm missing something but I don't undestand how StatefulSet relates to
the described functionality.
StatefulSet is more about persistence. Correct me if I'm wrong but my
current understanding is that we don't need to have any explicit state for a
thin client connection. I'd like this thing to be simple: if I'm working
with a pod and it fails then I just go to another one and try my request
again. The corner case is the best-effort affinity. As far as I understand it's not
a strict rule to communicate only directly with a pod running a node with a
primary partition. It's ok to fail-over in this case and communicate with
any pod. Am I right?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Pavel Tupitsyn]

Vladimir,

I agree with you, StatefulSet is not related here.

> it's not a strict rule to communicate only directly with a pod
> running a node with a primary partition
Yes, if a node with a primary partition is not known or can't be contacted,
we fail over to a default (random) node
(afaik this is how Java, C++ and .NET thin clients are implemented)

On Thu, Aug 13, 2020 at 12:44 PM Vladimir Pligin <[hidden email]>
wrote:

> Hi guys,
>
> Maybe I'm missing something but I don't undestand how StatefulSet relates
> to
> the described functionality.
> StatefulSet is more about persistence. Correct me if I'm wrong but my
> current understanding is that we don't need to have any explicit state for
> a
> thin client connection. I'd like this thing to be simple: if I'm working
> with a pod and it fails then I just go to another one and try my request
> again. The corner case is the best-effort affinity. As far as I can it's
> not
> a strict rule to communicate only directly with a pod running a node with a
> primary partition. It's ok to fail-over in this case and communicate with
> any pod. Am I right?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[Maksim Timonin]

Hi, StatefulSet provides predictable naming, then it should be easy to
configure a client with addresses ignite-1,ignite-2...ignite-N. So there is
no need in custom discovery, IPs etc. I think it corresponds to k8s
patterns, as some pods are different from others as they store specific
partitions (read, have a state). There will be some maintenance by a user -
the list of server namings should be provided too unless it's very simple
(ignite-XX).

As StatefulSet is required to enable persistence, I think it's not a big
problem to configure it the same way. And it should work out of the box.

I will investigate how much cost is to implement custom discovery for thin
clients. And compare it with the StatefulSet solution.

On Thu, Aug 13, 2020 at 12:52 PM Pavel Tupitsyn <[hidden email]>
wrote:

> Vladimir,
>
> I agree with you, StatefulSet is not related here.
>
> > it's not a strict rule to communicate only directly with a pod
> > running a node with a primary partition
> Yes, if a node with a primary partition is not known or can't be contacted,
> we fail over to a default (random) node
> (afaik this is how Java, C++ and .NET thin clients are implemented)
>
> On Thu, Aug 13, 2020 at 12:44 PM Vladimir Pligin <[hidden email]>
> wrote:
>
> > Hi guys,
> >
> > Maybe I'm missing something but I don't undestand how StatefulSet relates
> > to
> > the described functionality.
> > StatefulSet is more about persistence. Correct me if I'm wrong but my
> > current understanding is that we don't need to have any explicit state
> for
> > a
> > thin client connection. I'd like this thing to be simple: if I'm working
> > with a pod and it fails then I just go to another one and try my request
> > again. The corner case is the best-effort affinity. As far as I can it's
> > not
> > a strict rule to communicate only directly with a pod running a node
> with a
> > primary partition. It's ok to fail-over in this case and communicate with
> > any pod. Am I right?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

[dmagda]

Max,

There is no point in using DNS names instead of IP addresses if an
application developer stills needs to set the latter in a thin client
configuration.

The developer should just pass a single parameter, start the client and
forget about the connectivity matters. The state of the cluster will be
changing over the time - the nodes will join and go, but the developer
doesn’t need to touch the client’s settings or restart it to pass new DNS
names.

Check what we did with the Kubernetes IP finder for thick clients. It’s
trivial but reliable solution. My guts feel we should adopt it for thin
clients.

Denis

On Thursday, August 13, 2020, Max Timonin <[hidden email]> wrote:

> Hi, StatefulSet provides predictable naming, then it should be easy to
> configure a client with addresses ignite-1,ignite-2...ignite-N. So there is
> no need in custom discovery, IPs etc. I think it corresponds to k8s
> patterns, as some pods are different from others as they store specific
> partitions (read, have a state). There will be some maintenance by a user -
> the list of server namings should be provided too unless it's very simple
> (ignite-XX).
>
> As StatefulSet is required to enable persistence, I think it's not a big
> problem to configure it the same way. And it should work out of the box.
>
> I will investigate how much cost is to implement custom discovery for thin
> clients. And compare it with the StatefulSet solution.
>
> On Thu, Aug 13, 2020 at 12:52 PM Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Vladimir,
> >
> > I agree with you, StatefulSet is not related here.
> >
> > > it's not a strict rule to communicate only directly with a pod
> > > running a node with a primary partition
> > Yes, if a node with a primary partition is not known or can't be
> contacted,
> > we fail over to a default (random) node
> > (afaik this is how Java, C++ and .NET thin clients are implemented)
> >
> > On Thu, Aug 13, 2020 at 12:44 PM Vladimir Pligin <[hidden email]>
> > wrote:
> >
> > > Hi guys,
> > >
> > > Maybe I'm missing something but I don't undestand how StatefulSet
> relates
> > > to
> > > the described functionality.
> > > StatefulSet is more about persistence. Correct me if I'm wrong but my
> > > current understanding is that we don't need to have any explicit state
> > for
> > > a
> > > thin client connection. I'd like this thing to be simple: if I'm
> working
> > > with a pod and it fails then I just go to another one and try my
> request
> > > again. The corner case is the best-effort affinity. As far as I can
> it's
> > > not
> > > a strict rule to communicate only directly with a pod running a node
> > with a
> > > primary partition. It's ok to fail-over in this case and communicate
> with
> > > any pod. Am I right?
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>

--
-
Denis