Apache Ignite Developers - Legacy Mail Archive

IP Filtering in IPFinders

Classic

List

Threaded

16 messages Options

Atri Sharma-2

IP Filtering in IPFinders

Hi All,

I have opened the following JIRA for the said topic:

https://issues.apache.org/jira/browse/IGNITE-14606

The concept is to filter IPs based on a pattern or a blocklist in
IPFinders while consuming IPs. This is more pertinent for cloud based
IPFinders since they can have shared containers.

For the moment, I have implemented regex based filtering:

https://issues.apache.org/jira/browse/IGNITE-14607

for Azure Blob Storage IP Finder. Over time, we can extend the same to
other IP finders.

Please see the PR:

https://github.com/apache/ignite/pull/9024

Regards,

Atri

--
Regards,

Atri
Apache Concerted

Ilya Kasnacheev

Re: IP Filtering in IPFinders

Hello!

What are the expected use cases for this feature? Can you please elaborate?

Thanks,
--
Ilya Kasnacheev

ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:

> Hi All,
>
> I have opened the following JIRA for the said topic:
>
> https://issues.apache.org/jira/browse/IGNITE-14606
>
> The concept is to filter IPs based on a pattern or a blocklist in
> IPFinders while consuming IPs. This is more pertinent for cloud based
> IPFinders since they can have shared containers.
>
> For the moment, I have implemented regex based filtering:
>
> https://issues.apache.org/jira/browse/IGNITE-14607
>
> for Azure Blob Storage IP Finder. Over time, we can extend the same to
> other IP finders.
>
> Please see the PR:
>
> https://github.com/apache/ignite/pull/9024
>
> Regards,
>
> Atri
>
> --
> Regards,
>
> Atri
> Apache Concerted
>

Atri Sharma-2

Re: IP Filtering in IPFinders

Hi,

When a container is shared by multiple clusters, then this can be useful
for filtering IPs.

Also, things like VPC based barriers can be circumvented using this
technique.

On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]>
wrote:

> Hello!
>
> What are the expected use cases for this feature? Can you please elaborate?
>
> Thanks,
> --
> Ilya Kasnacheev
>
>
> ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
>
> > Hi All,
> >
> > I have opened the following JIRA for the said topic:
> >
> > https://issues.apache.org/jira/browse/IGNITE-14606
> >
> > The concept is to filter IPs based on a pattern or a blocklist in
> > IPFinders while consuming IPs. This is more pertinent for cloud based
> > IPFinders since they can have shared containers.
> >
> > For the moment, I have implemented regex based filtering:
> >
> > https://issues.apache.org/jira/browse/IGNITE-14607
> >
> > for Azure Blob Storage IP Finder. Over time, we can extend the same to
> > other IP finders.
> >
> > Please see the PR:
> >
> > https://github.com/apache/ignite/pull/9024
> >
> > Regards,
> >
> > Atri
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >
>

Valentin Kulichenko

Re: IP Filtering in IPFinders

Hi Atri,

Can you describe the scenario in a little more detail? What exactly do you
mean by a container shared by multiple clusters? What are the consequences
of this? How does the proposed solution solve the problem?

Also, I would suggest revisiting the design - I'm not sure such filtering
should be done on the IP finder level. Why not do this on the SPI level
instead? I would simply add something like "addressFilter" to the
TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will
be able to provide any implementations, including regex or anything else.

-Val

On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:

> Hi,
>
> When a container is shared by multiple clusters, then this can be useful
> for filtering IPs.
>
> Also, things like VPC based barriers can be circumvented using this
> technique.
>
> On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]>
> wrote:
>
> > Hello!
> >
> > What are the expected use cases for this feature? Can you please
> elaborate?
> >
> > Thanks,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> >
> > > Hi All,
> > >
> > > I have opened the following JIRA for the said topic:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-14606
> > >
> > > The concept is to filter IPs based on a pattern or a blocklist in
> > > IPFinders while consuming IPs. This is more pertinent for cloud based
> > > IPFinders since they can have shared containers.
> > >
> > > For the moment, I have implemented regex based filtering:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-14607
> > >
> > > for Azure Blob Storage IP Finder. Over time, we can extend the same to
> > > other IP finders.
> > >
> > > Please see the PR:
> > >
> > > https://github.com/apache/ignite/pull/9024
> > >
> > > Regards,
> > >
> > > Atri
> > >
> > > --
> > > Regards,
> > >
> > > Atri
> > > Apache Concerted
> > >
> >
>

Andrew Mashenkov

Re: IP Filtering in IPFinders

In reply to this post by Atri Sharma-2

Hi Atri,

You've added a new property to a base TcpDiscoveryIpFinder interface.
Actually, the only Azure IpFinder uses this setting, but the others.
This behavior may confuse the users.

Would you mind either making regexp filter setting a part of Azure IpFinder
only or fix other IpFinders as well?

On Wed, Apr 21, 2021 at 7:04 PM Atri Sharma <[hidden email]> wrote:

--
Best regards,
Andrey V. Mashenkov

Atri Sharma-2

Re: IP Filtering in IPFinders

In reply to this post by Valentin Kulichenko

Hi Val,

Consider a scenario where multiple Ignite clusters are running and for
operational ease (and also compliance, in some cases, e.g. to make
auditing easier), people can configure cloud based IP finders to share
the same container (blob container in Azure, S3 container in AWS etc).

In such a case, IPs for all clusters will be in the same container.
IPFinders of both the clusters will read the entire list. In this
case, address filtering will help ignore the irrelevant IP addresses.

Thank you for pointing me to the alternate direction. Let me research
that and revert.

Atri

On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
<[hidden email]> wrote:

>
> Hi Atri,
>
> Can you describe the scenario in a little more detail? What exactly do you
> mean by a container shared by multiple clusters? What are the consequences
> of this? How does the proposed solution solve the problem?
>
> Also, I would suggest revisiting the design - I'm not sure such filtering
> should be done on the IP finder level. Why not do this on the SPI level
> instead? I would simply add something like "addressFilter" to the
> TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will
> be able to provide any implementations, including regex or anything else.
>
> -Val
>
> On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:
>
> > Hi,
> >
> > When a container is shared by multiple clusters, then this can be useful
> > for filtering IPs.
> >
> > Also, things like VPC based barriers can be circumvented using this
> > technique.
> >
> > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]>
> > wrote:
> >
> > > Hello!
> > >
> > > What are the expected use cases for this feature? Can you please
> > elaborate?
> > >
> > > Thanks,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > >
> > > > Hi All,
> > > >
> > > > I have opened the following JIRA for the said topic:
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > >
> > > > The concept is to filter IPs based on a pattern or a blocklist in
> > > > IPFinders while consuming IPs. This is more pertinent for cloud based
> > > > IPFinders since they can have shared containers.
> > > >
> > > > For the moment, I have implemented regex based filtering:
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > >
> > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to
> > > > other IP finders.
> > > >
> > > > Please see the PR:
> > > >
> > > > https://github.com/apache/ignite/pull/9024
> > > >
> > > > Regards,
> > > >
> > > > Atri
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > Apache Concerted
> > > >
> > >
> >

--
Regards,

Atri
Apache Concerted

Atri Sharma-2

Re: IP Filtering in IPFinders

In reply to this post by Andrew Mashenkov

Hi Andrey,

Thanks for the message. Yes, that is the case since I wanted to show
the functionality to the group first.

Val has made a point about an alternate approach which seems cleaner
to me, so let me explore that. If we go that route, then we will not
need to change at IPFinder level which will make the change cleaner.

Atri

On Wed, Apr 21, 2021 at 10:50 PM Andrey Mashenkov
<[hidden email]> wrote:

>
> Hi Atri,
>
> You've added a new property to a base TcpDiscoveryIpFinder interface.
> Actually, the only Azure IpFinder uses this setting, but the others.
> This behavior may confuse the users.
>
> Would you mind either making regexp filter setting a part of Azure IpFinder
> only or fix other IpFinders as well?
>
>
> On Wed, Apr 21, 2021 at 7:04 PM Atri Sharma <[hidden email]> wrote:
>
> > Hi,
> >
> > When a container is shared by multiple clusters, then this can be useful
> > for filtering IPs.
> >
> > Also, things like VPC based barriers can be circumvented using this
> > technique.
> >
> > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]>
> > wrote:
> >
> > > Hello!
> > >
> > > What are the expected use cases for this feature? Can you please
> > elaborate?
> > >
> > > Thanks,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > >
> > > > Hi All,
> > > >
> > > > I have opened the following JIRA for the said topic:
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > >
> > > > The concept is to filter IPs based on a pattern or a blocklist in
> > > > IPFinders while consuming IPs. This is more pertinent for cloud based
> > > > IPFinders since they can have shared containers.
> > > >
> > > > For the moment, I have implemented regex based filtering:
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > >
> > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to
> > > > other IP finders.
> > > >
> > > > Please see the PR:
> > > >
> > > > https://github.com/apache/ignite/pull/9024
> > > >
> > > > Regards,
> > > >
> > > > Atri
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > Apache Concerted
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov

--
Regards,

Atri
Apache Concerted

Atri Sharma-2

Re: IP Filtering in IPFinders

In reply to this post by Atri Sharma-2

Another thing is IP addresses blocked by firewalls -- such IPs will
cause the cluster bootstrap to slow down.

On Thu, Apr 22, 2021 at 10:20 AM Atri Sharma <[hidden email]> wrote:

>
> Hi Val,
>
> Consider a scenario where multiple Ignite clusters are running and for
> operational ease (and also compliance, in some cases, e.g. to make
> auditing easier), people can configure cloud based IP finders to share
> the same container (blob container in Azure, S3 container in AWS etc).
>
> In such a case, IPs for all clusters will be in the same container.
> IPFinders of both the clusters will read the entire list. In this
> case, address filtering will help ignore the irrelevant IP addresses.
>
> Thank you for pointing me to the alternate direction. Let me research
> that and revert.
>
> Atri
>
> On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> <[hidden email]> wrote:
> >
> > Hi Atri,
> >
> > Can you describe the scenario in a little more detail? What exactly do you
> > mean by a container shared by multiple clusters? What are the consequences
> > of this? How does the proposed solution solve the problem?
> >
> > Also, I would suggest revisiting the design - I'm not sure such filtering
> > should be done on the IP finder level. Why not do this on the SPI level
> > instead? I would simply add something like "addressFilter" to the
> > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will
> > be able to provide any implementations, including regex or anything else.
> >
> > -Val
> >
> > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:
> >
> > > Hi,
> > >
> > > When a container is shared by multiple clusters, then this can be useful
> > > for filtering IPs.
> > >
> > > Also, things like VPC based barriers can be circumvented using this
> > > technique.
> > >
> > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]>
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > What are the expected use cases for this feature? Can you please
> > > elaborate?
> > > >
> > > > Thanks,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I have opened the following JIRA for the said topic:
> > > > >
> > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > >
> > > > > The concept is to filter IPs based on a pattern or a blocklist in
> > > > > IPFinders while consuming IPs. This is more pertinent for cloud based
> > > > > IPFinders since they can have shared containers.
> > > > >
> > > > > For the moment, I have implemented regex based filtering:
> > > > >
> > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > >
> > > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to
> > > > > other IP finders.
> > > > >
> > > > > Please see the PR:
> > > > >
> > > > > https://github.com/apache/ignite/pull/9024
> > > > >
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > > >
> > > >
> > >
>
> --
> Regards,
>
> Atri
> Apache Concerted

--
Regards,

Atri
Apache Concerted

Ilya Kasnacheev

Re: IP Filtering in IPFinders

In reply to this post by Atri Sharma-2

Hello!

AFAIK, a S3 container, Azure blob container, etc, is a relatively
lightweight entity, similar to a table in an SQL database. Why would
different clusters need to share the same discovery storage container?
When I tested Azure IP finder, it created several blob containers for me on
demand, based on the parameter passed to IP finder. If I wanted to have
more than one cluster it should have been seamless already.

I can theoretically see how address filtering may be useful to remove
public / private addresses or Docker gateway address, but it is usually
handled by setting localHost setting, although requiring tuning it for each
instance individually. Overall benefit seems to small.

This is why I am asking, do you have any specific scenario in mind where
this feature is an enabler? How did you arrive at the conclusion to go
forward with it?

Regards,
--
Ilya Kasnacheev

чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:

> Hi Val,
>
> Consider a scenario where multiple Ignite clusters are running and for
> operational ease (and also compliance, in some cases, e.g. to make
> auditing easier), people can configure cloud based IP finders to share
> the same container (blob container in Azure, S3 container in AWS etc).
>
> In such a case, IPs for all clusters will be in the same container.
> IPFinders of both the clusters will read the entire list. In this
> case, address filtering will help ignore the irrelevant IP addresses.
>
> Thank you for pointing me to the alternate direction. Let me research
> that and revert.
>
> Atri
>
> On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> <[hidden email]> wrote:
> >
> > Hi Atri,
> >
> > Can you describe the scenario in a little more detail? What exactly do
> you
> > mean by a container shared by multiple clusters? What are the
> consequences
> > of this? How does the proposed solution solve the problem?
> >
> > Also, I would suggest revisiting the design - I'm not sure such filtering
> > should be done on the IP finder level. Why not do this on the SPI level
> > instead? I would simply add something like "addressFilter" to the
> > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will
> > be able to provide any implementations, including regex or anything else.
> >
> > -Val
> >
> > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:
> >
> > > Hi,
> > >
> > > When a container is shared by multiple clusters, then this can be
> useful
> > > for filtering IPs.
> > >
> > > Also, things like VPC based barriers can be circumvented using this
> > > technique.
> > >
> > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]
> >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > What are the expected use cases for this feature? Can you please
> > > elaborate?
> > > >
> > > > Thanks,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I have opened the following JIRA for the said topic:
> > > > >
> > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > >
> > > > > The concept is to filter IPs based on a pattern or a blocklist in
> > > > > IPFinders while consuming IPs. This is more pertinent for cloud
> based
> > > > > IPFinders since they can have shared containers.
> > > > >
> > > > > For the moment, I have implemented regex based filtering:
> > > > >
> > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > >
> > > > > for Azure Blob Storage IP Finder. Over time, we can extend the
> same to
> > > > > other IP finders.
> > > > >
> > > > > Please see the PR:
> > > > >
> > > > > https://github.com/apache/ignite/pull/9024
> > > > >
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > > >
> > > >
> > >
>
> --
> Regards,
>
> Atri
> Apache Concerted
>

Atri Sharma-2

Re: IP Filtering in IPFinders

Hello!

I actually saw the shared container scenario being tried by somebody
who wanted an external script to monitor all IPs being used by his
clusters and hence thought of this idea. Another thing that came in
was the Firewall blocking a few IP addresses, hence the idea.

I feel that the footprint of this change is small, and can be useful
for esoteric use cases too without really interfering in any existing
code path. Val's suggestion seems the right way to go since it gives
the functionality without much change.

Thoughts?

On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
<[hidden email]> wrote:

>
> Hello!
>
> AFAIK, a S3 container, Azure blob container, etc, is a relatively
> lightweight entity, similar to a table in an SQL database. Why would
> different clusters need to share the same discovery storage container?
> When I tested Azure IP finder, it created several blob containers for me on
> demand, based on the parameter passed to IP finder. If I wanted to have
> more than one cluster it should have been seamless already.
>
> I can theoretically see how address filtering may be useful to remove
> public / private addresses or Docker gateway address, but it is usually
> handled by setting localHost setting, although requiring tuning it for each
> instance individually. Overall benefit seems to small.
>
> This is why I am asking, do you have any specific scenario in mind where
> this feature is an enabler? How did you arrive at the conclusion to go
> forward with it?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
>
> > Hi Val,
> >
> > Consider a scenario where multiple Ignite clusters are running and for
> > operational ease (and also compliance, in some cases, e.g. to make
> > auditing easier), people can configure cloud based IP finders to share
> > the same container (blob container in Azure, S3 container in AWS etc).
> >
> > In such a case, IPs for all clusters will be in the same container.
> > IPFinders of both the clusters will read the entire list. In this
> > case, address filtering will help ignore the irrelevant IP addresses.
> >
> > Thank you for pointing me to the alternate direction. Let me research
> > that and revert.
> >
> > Atri
> >
> > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > <[hidden email]> wrote:
> > >
> > > Hi Atri,
> > >
> > > Can you describe the scenario in a little more detail? What exactly do
> > you
> > > mean by a container shared by multiple clusters? What are the
> > consequences
> > > of this? How does the proposed solution solve the problem?
> > >
> > > Also, I would suggest revisiting the design - I'm not sure such filtering
> > > should be done on the IP finder level. Why not do this on the SPI level
> > > instead? I would simply add something like "addressFilter" to the
> > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will
> > > be able to provide any implementations, including regex or anything else.
> > >
> > > -Val
> > >
> > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > When a container is shared by multiple clusters, then this can be
> > useful
> > > > for filtering IPs.
> > > >
> > > > Also, things like VPC based barriers can be circumvented using this
> > > > technique.
> > > >
> > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <[hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > What are the expected use cases for this feature? Can you please
> > > > elaborate?
> > > > >
> > > > > Thanks,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I have opened the following JIRA for the said topic:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > >
> > > > > > The concept is to filter IPs based on a pattern or a blocklist in
> > > > > > IPFinders while consuming IPs. This is more pertinent for cloud
> > based
> > > > > > IPFinders since they can have shared containers.
> > > > > >
> > > > > > For the moment, I have implemented regex based filtering:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > >
> > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the
> > same to
> > > > > > other IP finders.
> > > > > >
> > > > > > Please see the PR:
> > > > > >
> > > > > > https://github.com/apache/ignite/pull/9024
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Atri
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Atri
> > > > > > Apache Concerted
> > > > > >
> > > > >
> > > >
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >

--
Regards,

Atri
Apache Concerted

Ilya Kasnacheev

Re: IP Filtering in IPFinders

Hello!

I'm still not fully convinced, but Val's approach sounds rational to me.

Regards,
--
Ilya Kasnacheev

чт, 22 апр. 2021 г. в 12:45, Atri Sharma <[hidden email]>:

> Hello!
>
> I actually saw the shared container scenario being tried by somebody
> who wanted an external script to monitor all IPs being used by his
> clusters and hence thought of this idea. Another thing that came in
> was the Firewall blocking a few IP addresses, hence the idea.
>
> I feel that the footprint of this change is small, and can be useful
> for esoteric use cases too without really interfering in any existing
> code path. Val's suggestion seems the right way to go since it gives
> the functionality without much change.
>
> Thoughts?
>
> On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> <[hidden email]> wrote:
> >
> > Hello!
> >
> > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > lightweight entity, similar to a table in an SQL database. Why would
> > different clusters need to share the same discovery storage container?
> > When I tested Azure IP finder, it created several blob containers for me
> on
> > demand, based on the parameter passed to IP finder. If I wanted to have
> > more than one cluster it should have been seamless already.
> >
> > I can theoretically see how address filtering may be useful to remove
> > public / private addresses or Docker gateway address, but it is usually
> > handled by setting localHost setting, although requiring tuning it for
> each
> > instance individually. Overall benefit seems to small.
> >
> > This is why I am asking, do you have any specific scenario in mind where
> > this feature is an enabler? How did you arrive at the conclusion to go
> > forward with it?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
> >
> > > Hi Val,
> > >
> > > Consider a scenario where multiple Ignite clusters are running and for
> > > operational ease (and also compliance, in some cases, e.g. to make
> > > auditing easier), people can configure cloud based IP finders to share
> > > the same container (blob container in Azure, S3 container in AWS etc).
> > >
> > > In such a case, IPs for all clusters will be in the same container.
> > > IPFinders of both the clusters will read the entire list. In this
> > > case, address filtering will help ignore the irrelevant IP addresses.
> > >
> > > Thank you for pointing me to the alternate direction. Let me research
> > > that and revert.
> > >
> > > Atri
> > >
> > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > <[hidden email]> wrote:
> > > >
> > > > Hi Atri,
> > > >
> > > > Can you describe the scenario in a little more detail? What exactly
> do
> > > you
> > > > mean by a container shared by multiple clusters? What are the
> > > consequences
> > > > of this? How does the proposed solution solve the problem?
> > > >
> > > > Also, I would suggest revisiting the design - I'm not sure such
> filtering
> > > > should be done on the IP finder level. Why not do this on the SPI
> level
> > > > instead? I would simply add something like "addressFilter" to the
> > > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you
> will
> > > > be able to provide any implementations, including regex or anything
> else.
> > > >
> > > > -Val
> > > >
> > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > When a container is shared by multiple clusters, then this can be
> > > useful
> > > > > for filtering IPs.
> > > > >
> > > > > Also, things like VPC based barriers can be circumvented using this
> > > > > technique.
> > > > >
> > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <
> [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > What are the expected use cases for this feature? Can you please
> > > > > elaborate?
> > > > > >
> > > > > > Thanks,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I have opened the following JIRA for the said topic:
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > > >
> > > > > > > The concept is to filter IPs based on a pattern or a blocklist
> in
> > > > > > > IPFinders while consuming IPs. This is more pertinent for cloud
> > > based
> > > > > > > IPFinders since they can have shared containers.
> > > > > > >
> > > > > > > For the moment, I have implemented regex based filtering:
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > > >
> > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the
> > > same to
> > > > > > > other IP finders.
> > > > > > >
> > > > > > > Please see the PR:
> > > > > > >
> > > > > > > https://github.com/apache/ignite/pull/9024
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Atri
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Atri
> > > > > > > Apache Concerted
> > > > > > >
> > > > > >
> > > > >
> > >
> > > --
> > > Regards,
> > >
> > > Atri
> > > Apache Concerted
> > >
>
> --
> Regards,
>
> Atri
> Apache Concerted
>

Atri Sharma-2

Re: IP Filtering in IPFinders

Hi Val and Ilya,

Thank you for taking the time to pursue this issue.

I have raised a new PR for the discussed approach. Please see and let
me know what you think:

https://github.com/apache/ignite/pull/9048

Regards,

Atri

On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
<[hidden email]> wrote:

>
> Hello!
>
> I'm still not fully convinced, but Val's approach sounds rational to me.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 22 апр. 2021 г. в 12:45, Atri Sharma <[hidden email]>:
>
> > Hello!
> >
> > I actually saw the shared container scenario being tried by somebody
> > who wanted an external script to monitor all IPs being used by his
> > clusters and hence thought of this idea. Another thing that came in
> > was the Firewall blocking a few IP addresses, hence the idea.
> >
> > I feel that the footprint of this change is small, and can be useful
> > for esoteric use cases too without really interfering in any existing
> > code path. Val's suggestion seems the right way to go since it gives
> > the functionality without much change.
> >
> > Thoughts?
> >
> > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > <[hidden email]> wrote:
> > >
> > > Hello!
> > >
> > > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > > lightweight entity, similar to a table in an SQL database. Why would
> > > different clusters need to share the same discovery storage container?
> > > When I tested Azure IP finder, it created several blob containers for me
> > on
> > > demand, based on the parameter passed to IP finder. If I wanted to have
> > > more than one cluster it should have been seamless already.
> > >
> > > I can theoretically see how address filtering may be useful to remove
> > > public / private addresses or Docker gateway address, but it is usually
> > > handled by setting localHost setting, although requiring tuning it for
> > each
> > > instance individually. Overall benefit seems to small.
> > >
> > > This is why I am asking, do you have any specific scenario in mind where
> > > this feature is an enabler? How did you arrive at the conclusion to go
> > > forward with it?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
> > >
> > > > Hi Val,
> > > >
> > > > Consider a scenario where multiple Ignite clusters are running and for
> > > > operational ease (and also compliance, in some cases, e.g. to make
> > > > auditing easier), people can configure cloud based IP finders to share
> > > > the same container (blob container in Azure, S3 container in AWS etc).
> > > >
> > > > In such a case, IPs for all clusters will be in the same container.
> > > > IPFinders of both the clusters will read the entire list. In this
> > > > case, address filtering will help ignore the irrelevant IP addresses.
> > > >
> > > > Thank you for pointing me to the alternate direction. Let me research
> > > > that and revert.
> > > >
> > > > Atri
> > > >
> > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > > <[hidden email]> wrote:
> > > > >
> > > > > Hi Atri,
> > > > >
> > > > > Can you describe the scenario in a little more detail? What exactly
> > do
> > > > you
> > > > > mean by a container shared by multiple clusters? What are the
> > > > consequences
> > > > > of this? How does the proposed solution solve the problem?
> > > > >
> > > > > Also, I would suggest revisiting the design - I'm not sure such
> > filtering
> > > > > should be done on the IP finder level. Why not do this on the SPI
> > level
> > > > > instead? I would simply add something like "addressFilter" to the
> > > > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you
> > will
> > > > > be able to provide any implementations, including regex or anything
> > else.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > When a container is shared by multiple clusters, then this can be
> > > > useful
> > > > > > for filtering IPs.
> > > > > >
> > > > > > Also, things like VPC based barriers can be circumvented using this
> > > > > > technique.
> > > > > >
> > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <
> > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello!
> > > > > > >
> > > > > > > What are the expected use cases for this feature? Can you please
> > > > > > elaborate?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > --
> > > > > > > Ilya Kasnacheev
> > > > > > >
> > > > > > >
> > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I have opened the following JIRA for the said topic:
> > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > > > >
> > > > > > > > The concept is to filter IPs based on a pattern or a blocklist
> > in
> > > > > > > > IPFinders while consuming IPs. This is more pertinent for cloud
> > > > based
> > > > > > > > IPFinders since they can have shared containers.
> > > > > > > >
> > > > > > > > For the moment, I have implemented regex based filtering:
> > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > > > >
> > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the
> > > > same to
> > > > > > > > other IP finders.
> > > > > > > >
> > > > > > > > Please see the PR:
> > > > > > > >
> > > > > > > > https://github.com/apache/ignite/pull/9024
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Atri
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Atri
> > > > > > > > Apache Concerted
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > Apache Concerted
> > > >
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >

--
Regards,

Atri
Apache Concerted

Valentin Kulichenko

Re: IP Filtering in IPFinders

Hi Atri,

I've noticed that you added the property to the IgniteConfiguration, but
it's applied only within the discovery. I feel like something is wrong here.

If this feature only relates to the discovery, then we should have the
configuration property on the TcpDiscoverySpi instead. Otherwise, the
filtering should be applied to all network components (although I'm not
necessarily sure what that would imply).

What do you think?

-Val

On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma <[hidden email]> wrote:

> Hi Val and Ilya,
>
> Thank you for taking the time to pursue this issue.
>
> I have raised a new PR for the discussed approach. Please see and let
> me know what you think:
>
> https://github.com/apache/ignite/pull/9048
>
> Regards,
>
> Atri
>
> On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
> <[hidden email]> wrote:
> >
> > Hello!
> >
> > I'm still not fully convinced, but Val's approach sounds rational to me.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 22 апр. 2021 г. в 12:45, Atri Sharma <[hidden email]>:
> >
> > > Hello!
> > >
> > > I actually saw the shared container scenario being tried by somebody
> > > who wanted an external script to monitor all IPs being used by his
> > > clusters and hence thought of this idea. Another thing that came in
> > > was the Firewall blocking a few IP addresses, hence the idea.
> > >
> > > I feel that the footprint of this change is small, and can be useful
> > > for esoteric use cases too without really interfering in any existing
> > > code path. Val's suggestion seems the right way to go since it gives
> > > the functionality without much change.
> > >
> > > Thoughts?
> > >
> > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > > <[hidden email]> wrote:
> > > >
> > > > Hello!
> > > >
> > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > > > lightweight entity, similar to a table in an SQL database. Why would
> > > > different clusters need to share the same discovery storage
> container?
> > > > When I tested Azure IP finder, it created several blob containers
> for me
> > > on
> > > > demand, based on the parameter passed to IP finder. If I wanted to
> have
> > > > more than one cluster it should have been seamless already.
> > > >
> > > > I can theoretically see how address filtering may be useful to remove
> > > > public / private addresses or Docker gateway address, but it is
> usually
> > > > handled by setting localHost setting, although requiring tuning it
> for
> > > each
> > > > instance individually. Overall benefit seems to small.
> > > >
> > > > This is why I am asking, do you have any specific scenario in mind
> where
> > > > this feature is an enabler? How did you arrive at the conclusion to
> go
> > > > forward with it?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
> > > >
> > > > > Hi Val,
> > > > >
> > > > > Consider a scenario where multiple Ignite clusters are running and
> for
> > > > > operational ease (and also compliance, in some cases, e.g. to make
> > > > > auditing easier), people can configure cloud based IP finders to
> share
> > > > > the same container (blob container in Azure, S3 container in AWS
> etc).
> > > > >
> > > > > In such a case, IPs for all clusters will be in the same container.
> > > > > IPFinders of both the clusters will read the entire list. In this
> > > > > case, address filtering will help ignore the irrelevant IP
> addresses.
> > > > >
> > > > > Thank you for pointing me to the alternate direction. Let me
> research
> > > > > that and revert.
> > > > >
> > > > > Atri
> > > > >
> > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > > > <[hidden email]> wrote:
> > > > > >
> > > > > > Hi Atri,
> > > > > >
> > > > > > Can you describe the scenario in a little more detail? What
> exactly
> > > do
> > > > > you
> > > > > > mean by a container shared by multiple clusters? What are the
> > > > > consequences
> > > > > > of this? How does the proposed solution solve the problem?
> > > > > >
> > > > > > Also, I would suggest revisiting the design - I'm not sure such
> > > filtering
> > > > > > should be done on the IP finder level. Why not do this on the SPI
> > > level
> > > > > > instead? I would simply add something like "addressFilter" to the
> > > > > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so
> you
> > > will
> > > > > > be able to provide any implementations, including regex or
> anything
> > > else.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]>
> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > When a container is shared by multiple clusters, then this can
> be
> > > > > useful
> > > > > > > for filtering IPs.
> > > > > > >
> > > > > > > Also, things like VPC based barriers can be circumvented using
> this
> > > > > > > technique.
> > > > > > >
> > > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <
> > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello!
> > > > > > > >
> > > > > > > > What are the expected use cases for this feature? Can you
> please
> > > > > > > elaborate?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > --
> > > > > > > > Ilya Kasnacheev
> > > > > > > >
> > > > > > > >
> > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > I have opened the following JIRA for the said topic:
> > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > > > > >
> > > > > > > > > The concept is to filter IPs based on a pattern or a
> blocklist
> > > in
> > > > > > > > > IPFinders while consuming IPs. This is more pertinent for
> cloud
> > > > > based
> > > > > > > > > IPFinders since they can have shared containers.
> > > > > > > > >
> > > > > > > > > For the moment, I have implemented regex based filtering:
> > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > > > > >
> > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend
> the
> > > > > same to
> > > > > > > > > other IP finders.
> > > > > > > > >
> > > > > > > > > Please see the PR:
> > > > > > > > >
> > > > > > > > > https://github.com/apache/ignite/pull/9024
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Atri
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Atri
> > > > > > > > > Apache Concerted
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > > >
> > >
> > > --
> > > Regards,
> > >
> > > Atri
> > > Apache Concerted
> > >
>
> --
> Regards,
>
> Atri
> Apache Concerted
>

Atri Sharma-2

Re: IP Filtering in IPFinders

Hi Ilya and Val,

Thank you for taking a look and providing insights. I have updated the
PR and raised another iteration.

Val, I have moved the configuration to TcpDiscoverySpi.

Please see and let me know your thoughts and comments.

Regards,

Atri

On Wed, Apr 28, 2021 at 2:11 AM Valentin Kulichenko
<[hidden email]> wrote:

>
> Hi Atri,
>
> I've noticed that you added the property to the IgniteConfiguration, but
> it's applied only within the discovery. I feel like something is wrong here.
>
> If this feature only relates to the discovery, then we should have the
> configuration property on the TcpDiscoverySpi instead. Otherwise, the
> filtering should be applied to all network components (although I'm not
> necessarily sure what that would imply).
>
> What do you think?
>
> -Val
>
> On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma <[hidden email]> wrote:
>
> > Hi Val and Ilya,
> >
> > Thank you for taking the time to pursue this issue.
> >
> > I have raised a new PR for the discussed approach. Please see and let
> > me know what you think:
> >
> > https://github.com/apache/ignite/pull/9048
> >
> > Regards,
> >
> > Atri
> >
> > On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
> > <[hidden email]> wrote:
> > >
> > > Hello!
> > >
> > > I'm still not fully convinced, but Val's approach sounds rational to me.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma <[hidden email]>:
> > >
> > > > Hello!
> > > >
> > > > I actually saw the shared container scenario being tried by somebody
> > > > who wanted an external script to monitor all IPs being used by his
> > > > clusters and hence thought of this idea. Another thing that came in
> > > > was the Firewall blocking a few IP addresses, hence the idea.
> > > >
> > > > I feel that the footprint of this change is small, and can be useful
> > > > for esoteric use cases too without really interfering in any existing
> > > > code path. Val's suggestion seems the right way to go since it gives
> > > > the functionality without much change.
> > > >
> > > > Thoughts?
> > > >
> > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > > > <[hidden email]> wrote:
> > > > >
> > > > > Hello!
> > > > >
> > > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > > > > lightweight entity, similar to a table in an SQL database. Why would
> > > > > different clusters need to share the same discovery storage
> > container?
> > > > > When I tested Azure IP finder, it created several blob containers
> > for me
> > > > on
> > > > > demand, based on the parameter passed to IP finder. If I wanted to
> > have
> > > > > more than one cluster it should have been seamless already.
> > > > >
> > > > > I can theoretically see how address filtering may be useful to remove
> > > > > public / private addresses or Docker gateway address, but it is
> > usually
> > > > > handled by setting localHost setting, although requiring tuning it
> > for
> > > > each
> > > > > instance individually. Overall benefit seems to small.
> > > > >
> > > > > This is why I am asking, do you have any specific scenario in mind
> > where
> > > > > this feature is an enabler? How did you arrive at the conclusion to
> > go
> > > > > forward with it?
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
> > > > >
> > > > > > Hi Val,
> > > > > >
> > > > > > Consider a scenario where multiple Ignite clusters are running and
> > for
> > > > > > operational ease (and also compliance, in some cases, e.g. to make
> > > > > > auditing easier), people can configure cloud based IP finders to
> > share
> > > > > > the same container (blob container in Azure, S3 container in AWS
> > etc).
> > > > > >
> > > > > > In such a case, IPs for all clusters will be in the same container.
> > > > > > IPFinders of both the clusters will read the entire list. In this
> > > > > > case, address filtering will help ignore the irrelevant IP
> > addresses.
> > > > > >
> > > > > > Thank you for pointing me to the alternate direction. Let me
> > research
> > > > > > that and revert.
> > > > > >
> > > > > > Atri
> > > > > >
> > > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > > > > <[hidden email]> wrote:
> > > > > > >
> > > > > > > Hi Atri,
> > > > > > >
> > > > > > > Can you describe the scenario in a little more detail? What
> > exactly
> > > > do
> > > > > > you
> > > > > > > mean by a container shared by multiple clusters? What are the
> > > > > > consequences
> > > > > > > of this? How does the proposed solution solve the problem?
> > > > > > >
> > > > > > > Also, I would suggest revisiting the design - I'm not sure such
> > > > filtering
> > > > > > > should be done on the IP finder level. Why not do this on the SPI
> > > > level
> > > > > > > instead? I would simply add something like "addressFilter" to the
> > > > > > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so
> > you
> > > > will
> > > > > > > be able to provide any implementations, including regex or
> > anything
> > > > else.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]>
> > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > When a container is shared by multiple clusters, then this can
> > be
> > > > > > useful
> > > > > > > > for filtering IPs.
> > > > > > > >
> > > > > > > > Also, things like VPC based barriers can be circumvented using
> > this
> > > > > > > > technique.
> > > > > > > >
> > > > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <
> > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello!
> > > > > > > > >
> > > > > > > > > What are the expected use cases for this feature? Can you
> > please
> > > > > > > > elaborate?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > --
> > > > > > > > > Ilya Kasnacheev
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <[hidden email]>:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > I have opened the following JIRA for the said topic:
> > > > > > > > > >
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > > > > > >
> > > > > > > > > > The concept is to filter IPs based on a pattern or a
> > blocklist
> > > > in
> > > > > > > > > > IPFinders while consuming IPs. This is more pertinent for
> > cloud
> > > > > > based
> > > > > > > > > > IPFinders since they can have shared containers.
> > > > > > > > > >
> > > > > > > > > > For the moment, I have implemented regex based filtering:
> > > > > > > > > >
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > > > > > >
> > > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend
> > the
> > > > > > same to
> > > > > > > > > > other IP finders.
> > > > > > > > > >
> > > > > > > > > > Please see the PR:
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/ignite/pull/9024
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Atri
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Atri
> > > > > > > > > > Apache Concerted
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Atri
> > > > > > Apache Concerted
> > > > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > Apache Concerted
> > > >
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >

--
Regards,

Atri
Apache Concerted

Valentin Kulichenko

Re: IP Filtering in IPFinders

I'm OK with the design.

Ilya, please feel free to merge if the implementation and tests look good
to you.

-Val

On Wed, Apr 28, 2021 at 1:07 AM Atri Sharma <[hidden email]> wrote:

> Hi Ilya and Val,
>
> Thank you for taking a look and providing insights. I have updated the
> PR and raised another iteration.
>
> Val, I have moved the configuration to TcpDiscoverySpi.
>
> Please see and let me know your thoughts and comments.
>
> Regards,
>
> Atri
>
> On Wed, Apr 28, 2021 at 2:11 AM Valentin Kulichenko
> <[hidden email]> wrote:
> >
> > Hi Atri,
> >
> > I've noticed that you added the property to the IgniteConfiguration, but
> > it's applied only within the discovery. I feel like something is wrong
> here.
> >
> > If this feature only relates to the discovery, then we should have the
> > configuration property on the TcpDiscoverySpi instead. Otherwise, the
> > filtering should be applied to all network components (although I'm not
> > necessarily sure what that would imply).
> >
> > What do you think?
> >
> > -Val
> >
> > On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma <[hidden email]> wrote:
> >
> > > Hi Val and Ilya,
> > >
> > > Thank you for taking the time to pursue this issue.
> > >
> > > I have raised a new PR for the discussed approach. Please see and let
> > > me know what you think:
> > >
> > > https://github.com/apache/ignite/pull/9048
> > >
> > > Regards,
> > >
> > > Atri
> > >
> > > On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
> > > <[hidden email]> wrote:
> > > >
> > > > Hello!
> > > >
> > > > I'm still not fully convinced, but Val's approach sounds rational to
> me.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma <[hidden email]>:
> > > >
> > > > > Hello!
> > > > >
> > > > > I actually saw the shared container scenario being tried by
> somebody
> > > > > who wanted an external script to monitor all IPs being used by his
> > > > > clusters and hence thought of this idea. Another thing that came in
> > > > > was the Firewall blocking a few IP addresses, hence the idea.
> > > > >
> > > > > I feel that the footprint of this change is small, and can be
> useful
> > > > > for esoteric use cases too without really interfering in any
> existing
> > > > > code path. Val's suggestion seems the right way to go since it
> gives
> > > > > the functionality without much change.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > > > > <[hidden email]> wrote:
> > > > > >
> > > > > > Hello!
> > > > > >
> > > > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > > > > > lightweight entity, similar to a table in an SQL database. Why
> would
> > > > > > different clusters need to share the same discovery storage
> > > container?
> > > > > > When I tested Azure IP finder, it created several blob containers
> > > for me
> > > > > on
> > > > > > demand, based on the parameter passed to IP finder. If I wanted
> to
> > > have
> > > > > > more than one cluster it should have been seamless already.
> > > > > >
> > > > > > I can theoretically see how address filtering may be useful to
> remove
> > > > > > public / private addresses or Docker gateway address, but it is
> > > usually
> > > > > > handled by setting localHost setting, although requiring tuning
> it
> > > for
> > > > > each
> > > > > > instance individually. Overall benefit seems to small.
> > > > > >
> > > > > > This is why I am asking, do you have any specific scenario in
> mind
> > > where
> > > > > > this feature is an enabler? How did you arrive at the conclusion
> to
> > > go
> > > > > > forward with it?
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
> > > > > >
> > > > > > > Hi Val,
> > > > > > >
> > > > > > > Consider a scenario where multiple Ignite clusters are running
> and
> > > for
> > > > > > > operational ease (and also compliance, in some cases, e.g. to
> make
> > > > > > > auditing easier), people can configure cloud based IP finders
> to
> > > share
> > > > > > > the same container (blob container in Azure, S3 container in
> AWS
> > > etc).
> > > > > > >
> > > > > > > In such a case, IPs for all clusters will be in the same
> container.
> > > > > > > IPFinders of both the clusters will read the entire list. In
> this
> > > > > > > case, address filtering will help ignore the irrelevant IP
> > > addresses.
> > > > > > >
> > > > > > > Thank you for pointing me to the alternate direction. Let me
> > > research
> > > > > > > that and revert.
> > > > > > >
> > > > > > > Atri
> > > > > > >
> > > > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > > > > > <[hidden email]> wrote:
> > > > > > > >
> > > > > > > > Hi Atri,
> > > > > > > >
> > > > > > > > Can you describe the scenario in a little more detail? What
> > > exactly
> > > > > do
> > > > > > > you
> > > > > > > > mean by a container shared by multiple clusters? What are the
> > > > > > > consequences
> > > > > > > > of this? How does the proposed solution solve the problem?
> > > > > > > >
> > > > > > > > Also, I would suggest revisiting the design - I'm not sure
> such
> > > > > filtering
> > > > > > > > should be done on the IP finder level. Why not do this on
> the SPI
> > > > > level
> > > > > > > > instead? I would simply add something like "addressFilter"
> to the
> > > > > > > > TcpDiscoverySpi. The filter can be a generic
> IgnitePredicate, so
> > > you
> > > > > will
> > > > > > > > be able to provide any implementations, including regex or
> > > anything
> > > > > else.
> > > > > > > >
> > > > > > > > -Val
> > > > > > > >
> > > > > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <[hidden email]
> >
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > When a container is shared by multiple clusters, then this
> can
> > > be
> > > > > > > useful
> > > > > > > > > for filtering IPs.
> > > > > > > > >
> > > > > > > > > Also, things like VPC based barriers can be circumvented
> using
> > > this
> > > > > > > > > technique.
> > > > > > > > >
> > > > > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <
> > > > > [hidden email]
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello!
> > > > > > > > > >
> > > > > > > > > > What are the expected use cases for this feature? Can you
> > > please
> > > > > > > > > elaborate?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > --
> > > > > > > > > > Ilya Kasnacheev
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <
> [hidden email]>:
> > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > I have opened the following JIRA for the said topic:
> > > > > > > > > > >
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > > > > > > >
> > > > > > > > > > > The concept is to filter IPs based on a pattern or a
> > > blocklist
> > > > > in
> > > > > > > > > > > IPFinders while consuming IPs. This is more pertinent
> for
> > > cloud
> > > > > > > based
> > > > > > > > > > > IPFinders since they can have shared containers.
> > > > > > > > > > >
> > > > > > > > > > > For the moment, I have implemented regex based
> filtering:
> > > > > > > > > > >
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > > > > > > >
> > > > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can
> extend
> > > the
> > > > > > > same to
> > > > > > > > > > > other IP finders.
> > > > > > > > > > >
> > > > > > > > > > > Please see the PR:
> > > > > > > > > > >
> > > > > > > > > > > https://github.com/apache/ignite/pull/9024
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Atri
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Atri
> > > > > > > > > > > Apache Concerted
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Atri
> > > > > > > Apache Concerted
> > > > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > > >
> > >
> > > --
> > > Regards,
> > >
> > > Atri
> > > Apache Concerted
> > >
>
> --
> Regards,
>
> Atri
> Apache Concerted
>

Atri Sharma-2

Re: IP Filtering in IPFinders

Thank you, Val.

Ilya, please let me know if the PR looks ok.

On Thu, 29 Apr 2021, 00:19 Valentin Kulichenko, <
[hidden email]> wrote:

> I'm OK with the design.
>
> Ilya, please feel free to merge if the implementation and tests look good
> to you.
>
> -Val
>
> On Wed, Apr 28, 2021 at 1:07 AM Atri Sharma <[hidden email]> wrote:
>
> > Hi Ilya and Val,
> >
> > Thank you for taking a look and providing insights. I have updated the
> > PR and raised another iteration.
> >
> > Val, I have moved the configuration to TcpDiscoverySpi.
> >
> > Please see and let me know your thoughts and comments.
> >
> > Regards,
> >
> > Atri
> >
> > On Wed, Apr 28, 2021 at 2:11 AM Valentin Kulichenko
> > <[hidden email]> wrote:
> > >
> > > Hi Atri,
> > >
> > > I've noticed that you added the property to the IgniteConfiguration,
> but
> > > it's applied only within the discovery. I feel like something is wrong
> > here.
> > >
> > > If this feature only relates to the discovery, then we should have the
> > > configuration property on the TcpDiscoverySpi instead. Otherwise, the
> > > filtering should be applied to all network components (although I'm not
> > > necessarily sure what that would imply).
> > >
> > > What do you think?
> > >
> > > -Val
> > >
> > > On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma <[hidden email]> wrote:
> > >
> > > > Hi Val and Ilya,
> > > >
> > > > Thank you for taking the time to pursue this issue.
> > > >
> > > > I have raised a new PR for the discussed approach. Please see and let
> > > > me know what you think:
> > > >
> > > > https://github.com/apache/ignite/pull/9048
> > > >
> > > > Regards,
> > > >
> > > > Atri
> > > >
> > > > On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
> > > > <[hidden email]> wrote:
> > > > >
> > > > > Hello!
> > > > >
> > > > > I'm still not fully convinced, but Val's approach sounds rational
> to
> > me.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma <[hidden email]>:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > I actually saw the shared container scenario being tried by
> > somebody
> > > > > > who wanted an external script to monitor all IPs being used by
> his
> > > > > > clusters and hence thought of this idea. Another thing that came
> in
> > > > > > was the Firewall blocking a few IP addresses, hence the idea.
> > > > > >
> > > > > > I feel that the footprint of this change is small, and can be
> > useful
> > > > > > for esoteric use cases too without really interfering in any
> > existing
> > > > > > code path. Val's suggestion seems the right way to go since it
> > gives
> > > > > > the functionality without much change.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > > > > > <[hidden email]> wrote:
> > > > > > >
> > > > > > > Hello!
> > > > > > >
> > > > > > > AFAIK, a S3 container, Azure blob container, etc, is a
> relatively
> > > > > > > lightweight entity, similar to a table in an SQL database. Why
> > would
> > > > > > > different clusters need to share the same discovery storage
> > > > container?
> > > > > > > When I tested Azure IP finder, it created several blob
> containers
> > > > for me
> > > > > > on
> > > > > > > demand, based on the parameter passed to IP finder. If I wanted
> > to
> > > > have
> > > > > > > more than one cluster it should have been seamless already.
> > > > > > >
> > > > > > > I can theoretically see how address filtering may be useful to
> > remove
> > > > > > > public / private addresses or Docker gateway address, but it is
> > > > usually
> > > > > > > handled by setting localHost setting, although requiring tuning
> > it
> > > > for
> > > > > > each
> > > > > > > instance individually. Overall benefit seems to small.
> > > > > > >
> > > > > > > This is why I am asking, do you have any specific scenario in
> > mind
> > > > where
> > > > > > > this feature is an enabler? How did you arrive at the
> conclusion
> > to
> > > > go
> > > > > > > forward with it?
> > > > > > >
> > > > > > > Regards,
> > > > > > > --
> > > > > > > Ilya Kasnacheev
> > > > > > >
> > > > > > >
> > > > > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma <[hidden email]>:
> > > > > > >
> > > > > > > > Hi Val,
> > > > > > > >
> > > > > > > > Consider a scenario where multiple Ignite clusters are
> running
> > and
> > > > for
> > > > > > > > operational ease (and also compliance, in some cases, e.g. to
> > make
> > > > > > > > auditing easier), people can configure cloud based IP finders
> > to
> > > > share
> > > > > > > > the same container (blob container in Azure, S3 container in
> > AWS
> > > > etc).
> > > > > > > >
> > > > > > > > In such a case, IPs for all clusters will be in the same
> > container.
> > > > > > > > IPFinders of both the clusters will read the entire list. In
> > this
> > > > > > > > case, address filtering will help ignore the irrelevant IP
> > > > addresses.
> > > > > > > >
> > > > > > > > Thank you for pointing me to the alternate direction. Let me
> > > > research
> > > > > > > > that and revert.
> > > > > > > >
> > > > > > > > Atri
> > > > > > > >
> > > > > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > > > > > > <[hidden email]> wrote:
> > > > > > > > >
> > > > > > > > > Hi Atri,
> > > > > > > > >
> > > > > > > > > Can you describe the scenario in a little more detail? What
> > > > exactly
> > > > > > do
> > > > > > > > you
> > > > > > > > > mean by a container shared by multiple clusters? What are
> the
> > > > > > > > consequences
> > > > > > > > > of this? How does the proposed solution solve the problem?
> > > > > > > > >
> > > > > > > > > Also, I would suggest revisiting the design - I'm not sure
> > such
> > > > > > filtering
> > > > > > > > > should be done on the IP finder level. Why not do this on
> > the SPI
> > > > > > level
> > > > > > > > > instead? I would simply add something like "addressFilter"
> > to the
> > > > > > > > > TcpDiscoverySpi. The filter can be a generic
> > IgnitePredicate, so
> > > > you
> > > > > > will
> > > > > > > > > be able to provide any implementations, including regex or
> > > > anything
> > > > > > else.
> > > > > > > > >
> > > > > > > > > -Val
> > > > > > > > >
> > > > > > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma <
> [hidden email]
> > >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > When a container is shared by multiple clusters, then
> this
> > can
> > > > be
> > > > > > > > useful
> > > > > > > > > > for filtering IPs.
> > > > > > > > > >
> > > > > > > > > > Also, things like VPC based barriers can be circumvented
> > using
> > > > this
> > > > > > > > > > technique.
> > > > > > > > > >
> > > > > > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, <
> > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello!
> > > > > > > > > > >
> > > > > > > > > > > What are the expected use cases for this feature? Can
> you
> > > > please
> > > > > > > > > > elaborate?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > --
> > > > > > > > > > > Ilya Kasnacheev
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma <
> > [hidden email]>:
> > > > > > > > > > >
> > > > > > > > > > > > Hi All,
> > > > > > > > > > > >
> > > > > > > > > > > > I have opened the following JIRA for the said topic:
> > > > > > > > > > > >
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606
> > > > > > > > > > > >
> > > > > > > > > > > > The concept is to filter IPs based on a pattern or a
> > > > blocklist
> > > > > > in
> > > > > > > > > > > > IPFinders while consuming IPs. This is more pertinent
> > for
> > > > cloud
> > > > > > > > based
> > > > > > > > > > > > IPFinders since they can have shared containers.
> > > > > > > > > > > >
> > > > > > > > > > > > For the moment, I have implemented regex based
> > filtering:
> > > > > > > > > > > >
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607
> > > > > > > > > > > >
> > > > > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can
> > extend
> > > > the
> > > > > > > > same to
> > > > > > > > > > > > other IP finders.
> > > > > > > > > > > >
> > > > > > > > > > > > Please see the PR:
> > > > > > > > > > > >
> > > > > > > > > > > > https://github.com/apache/ignite/pull/9024
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Atri
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Atri
> > > > > > > > > > > > Apache Concerted
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Atri
> > > > > > > > Apache Concerted
> > > > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Atri
> > > > > > Apache Concerted
> > > > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > Apache Concerted
> > > >
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >
>