Merging all network components to a single one

[yzhdanov]

Guys,

I have an idea of merging all net components to one.

Now we have the following components interacting via network:
1. discovery
2. communication
3. rest
4. odbc
5. ignite-hadoop
6. time processor (being removed together with clock mode)
7. IPC communication endpoint

2-6 use GridNioServer each with different set of selector threads which may
result to exceeding the number of cores. Tcp discovery uses blocking socket
API.

All above mean that we may require many TCP ports opened on nodes. When it
comes to some secured environments with firewalls and gathering special
permissions to open new ports Ignite installation may become painful.

What if we have the only TCP port per node (of course we can still bind all
the components to different ports) and single component that encapsulates
all the network activities and resource management? All components that
need network interaction may register filter chain to the network component
and start getting/sending network messages.

In other words, I suggest to have a single set of nio-selectors and clean
API to install network listeners to satisfy demands of all other
components. E.g. discovery, communication and rest will not open their own
servers but go to the new NetworkProcessor and setup listeners chain on
some ports (possibly on the default one and NetworkProcessor will properly
dispatch incoming connections between the components)

Current implementation has the following drawbacks that will be fixed by
new approach.
1. may require too many ports opened
2. may have selector threads count exceeding number of CPU which may lead
to performance degradation.

Please share your thoughts or ask questions.

--Yakov

[voipp]

What tasks do discovery component responsible for ?

вт, 7 мар. 2017 г. в 13:15, Yakov Zhdanov <[hidden email]>:

> Guys,
>
> I have an idea of merging all net components to one.
>
> Now we have the following components interacting via network:
> 1. discovery
> 2. communication
> 3. rest
> 4. odbc
> 5. ignite-hadoop
> 6. time processor (being removed together with clock mode)
> 7. IPC communication endpoint
>
> 2-6 use GridNioServer each with different set of selector threads which may
> result to exceeding the number of cores. Tcp discovery uses blocking socket
> API.
>
> All above mean that we may require many TCP ports opened on nodes. When it
> comes to some secured environments with firewalls and gathering special
> permissions to open new ports Ignite installation may become painful.
>
> What if we have the only TCP port per node (of course we can still bind all
> the components to different ports) and single component that encapsulates
> all the network activities and resource management? All components that
> need network interaction may register filter chain to the network component
> and start getting/sending network messages.
>
> In other words, I suggest to have a single set of nio-selectors and clean
> API to install network listeners to satisfy demands of all other
> components. E.g. discovery, communication and rest will not open their own
> servers but go to the new NetworkProcessor and setup listeners chain on
> some ports (possibly on the default one and NetworkProcessor will properly
> dispatch incoming connections between the components)
>
> Current implementation has the following drawbacks that will be fixed by
> new approach.
> 1. may require too many ports opened
> 2. may have selector threads count exceeding number of CPU which may lead
> to performance degradation.
>
> Please share your thoughts or ask questions.
>
> --Yakov
>

--

*Best Regards,*

*Kuznetsov Aleksey*

[dsetrakyan]

Yakov,

I think you are proposing to have a single NIO (or TCP) SPI and have all
other SPIs and components register with it and receive message callbacks.
Is that right? If yes, then I really like the idea.

D.

On Tue, Mar 7, 2017 at 2:15 AM, Yakov Zhdanov <[hidden email]> wrote:

> Guys,
>
> I have an idea of merging all net components to one.
>
> Now we have the following components interacting via network:
> 1. discovery
> 2. communication
> 3. rest
> 4. odbc
> 5. ignite-hadoop
> 6. time processor (being removed together with clock mode)
> 7. IPC communication endpoint
>
> 2-6 use GridNioServer each with different set of selector threads which may
> result to exceeding the number of cores. Tcp discovery uses blocking socket
> API.
>
> All above mean that we may require many TCP ports opened on nodes. When it
> comes to some secured environments with firewalls and gathering special
> permissions to open new ports Ignite installation may become painful.
>
> What if we have the only TCP port per node (of course we can still bind all
> the components to different ports) and single component that encapsulates
> all the network activities and resource management? All components that
> need network interaction may register filter chain to the network component
> and start getting/sending network messages.
>
> In other words, I suggest to have a single set of nio-selectors and clean
> API to install network listeners to satisfy demands of all other
> components. E.g. discovery, communication and rest will not open their own
> servers but go to the new NetworkProcessor and setup listeners chain on
> some ports (possibly on the default one and NetworkProcessor will properly
> dispatch incoming connections between the components)
>
> Current implementation has the following drawbacks that will be fixed by
> new approach.
> 1. may require too many ports opened
> 2. may have selector threads count exceeding number of CPU which may lead
> to performance degradation.
>
> Please share your thoughts or ask questions.
>
> --Yakov
>

[dmagda]

Personally, I’m fully for this idea.

BTW, there is already a ticket for this task created by Yakov some time ago:
https://issues.apache.org/jira/browse/IGNITE-3480 <https://issues.apache.org/jira/browse/IGNITE-3480>

—
Denis

> On Mar 7, 2017, at 10:50 AM, Dmitriy Setrakyan <[hidden email]> wrote:
>
> Yakov,
>
> I think you are proposing to have a single NIO (or TCP) SPI and have all
> other SPIs and components register with it and receive message callbacks.
> Is that right? If yes, then I really like the idea.
>
> D.
>
> On Tue, Mar 7, 2017 at 2:15 AM, Yakov Zhdanov <[hidden email]> wrote:
>
>> Guys,
>>
>> I have an idea of merging all net components to one.
>>
>> Now we have the following components interacting via network:
>> 1. discovery
>> 2. communication
>> 3. rest
>> 4. odbc
>> 5. ignite-hadoop
>> 6. time processor (being removed together with clock mode)
>> 7. IPC communication endpoint
>>
>> 2-6 use GridNioServer each with different set of selector threads which may
>> result to exceeding the number of cores. Tcp discovery uses blocking socket
>> API.
>>
>> All above mean that we may require many TCP ports opened on nodes. When it
>> comes to some secured environments with firewalls and gathering special
>> permissions to open new ports Ignite installation may become painful.
>>
>> What if we have the only TCP port per node (of course we can still bind all
>> the components to different ports) and single component that encapsulates
>> all the network activities and resource management? All components that
>> need network interaction may register filter chain to the network component
>> and start getting/sending network messages.
>>
>> In other words, I suggest to have a single set of nio-selectors and clean
>> API to install network listeners to satisfy demands of all other
>> components. E.g. discovery, communication and rest will not open their own
>> servers but go to the new NetworkProcessor and setup listeners chain on
>> some ports (possibly on the default one and NetworkProcessor will properly
>> dispatch incoming connections between the components)
>>
>> Current implementation has the following drawbacks that will be fixed by
>> new approach.
>> 1. may require too many ports opened
>> 2. may have selector threads count exceeding number of CPU which may lead
>> to performance degradation.
>>
>> Please share your thoughts or ask questions.
>>
>> --Yakov
>>

[yzhdanov]

Dmitry, yes, your understanding is correct.

Denis, I remember I already shared the idea a while ago, but completely
forgot about the ticket. Thanks for bringing it up! You should give me
couple of tips on how to use "Search" function in Jira :)

Guys, this will be compatibility breaking change, as there will be no, for
example, discovery and communication SPIs. What if we move current
implementations to internal packages, introduce configuration
properties/beans so we can release and continue development in 2.0 without
breaking compatibility?

--Yakov

[dmagda]

Yakov,

However an end user still will be able to provide its own implementation for this combiner discovery & communication layers, right?
Just want to be sure that if I want to add some custom RDMA, RoCe or InfiniBand implementation then I’ll be able to do this.

—
Denis

> On Mar 9, 2017, at 1:11 AM, Yakov Zhdanov <[hidden email]> wrote:
>
> Dmitry, yes, your understanding is correct.
>
> Denis, I remember I already shared the idea a while ago, but completely
> forgot about the ticket. Thanks for bringing it up! You should give me
> couple of tips on how to use "Search" function in Jira :)
>
> Guys, this will be compatibility breaking change, as there will be no, for
> example, discovery and communication SPIs. What if we move current
> implementations to internal packages, introduce configuration
> properties/beans so we can release and continue development in 2.0 without
> breaking compatibility?
>
> --Yakov

[dsetrakyan]

To my understanding, Yakov is suggesting that we simply have one very low
level SPI - let's call it IgniteTcpSpi and current implementations of
discovery and communication will be removed from SPI layer and will
register callbacks with this new SPI.

I think this is too big of a change to try to squeeze it into 2.0 release.

Yakov, what are your thoughts.

On Thu, Mar 9, 2017 at 10:30 AM, Denis Magda <[hidden email]> wrote:

> Yakov,
>
> However an end user still will be able to provide its own implementation
> for this combiner discovery & communication layers, right?
> Just want to be sure that if I want to add some custom RDMA, RoCe or
> InfiniBand implementation then I’ll be able to do this.
>
> —
> Denis
>
> > On Mar 9, 2017, at 1:11 AM, Yakov Zhdanov <[hidden email]> wrote:
> >
> > Dmitry, yes, your understanding is correct.
> >
> > Denis, I remember I already shared the idea a while ago, but completely
> > forgot about the ticket. Thanks for bringing it up! You should give me
> > couple of tips on how to use "Search" function in Jira :)
> >
> > Guys, this will be compatibility breaking change, as there will be no,
> for
> > example, discovery and communication SPIs. What if we move current
> > implementations to internal packages, introduce configuration
> > properties/beans so we can release and continue development in 2.0
> without
> > breaking compatibility?
> >
> > --Yakov
>
>

[yzhdanov]

Dmitry, you are right - this seems to be a very complex change. However, we
can we move current implementations to internal packages, introduce
configuration properties/beans so we can release and continue development
in 2.0 without breaking compatibility.

Denis, I think we can allow this as well, but not as SPI since I want to
drop them and create 1 internal component to manage all interconnections. I
think this is enough since additions/changes will be rare and they will
result in changing internal code (including support of technologies you
mentioned). None of our users implemented discovery or communication on
their own, and I doubt if someone ever will. So, what is the purpose of
keeping SPIs?

--Yakov

[dmagda]

> Denis, I think we can allow this as well, but not as SPI since I want to
> drop them and create 1 internal component to manage all interconnections. I
> think this is enough since additions/changes will be rare and they will
> result in changing internal code (including support of technologies you
> mentioned). None of our users implemented discovery or communication on
> their own, and I doubt if someone ever will. So, what is the purpose of
> keeping SPIs?

Yasha, I don’t insist to keep the SPIs but rather want to clarify that it will be feasible to provide a custom implementation of this new networking component if needed. Looks like this will be possible.

> Dmitry, you are right - this seems to be a very complex change. However, we
> can we move current implementations to internal packages, introduce
> configuration properties/beans so we can release and continue development
> in 2.0 without breaking compatibility.

Let’s do this if there is a one who can take over the interface design and complete it by 2.0 release.

—
Denis

> On Mar 10, 2017, at 7:39 AM, Yakov Zhdanov <[hidden email]> wrote:
>
> Dmitry, you are right - this seems to be a very complex change. However, we
> can we move current implementations to internal packages, introduce
> configuration properties/beans so we can release and continue development
> in 2.0 without breaking compatibility.
>
> Denis, I think we can allow this as well, but not as SPI since I want to
> drop them and create 1 internal component to manage all interconnections. I
> think this is enough since additions/changes will be rare and they will
> result in changing internal code (including support of technologies you
> mentioned). None of our users implemented discovery or communication on
> their own, and I doubt if someone ever will. So, what is the purpose of
> keeping SPIs?
>
> --Yakov

[dsetrakyan]

Agree with Denis. It is not practical to remove any existing SPIs unless we
come up with proper TCP SPI design and implementation.

I think it will be enough to simply deprecate the existing SPIs we plan to
remove, without physically removing them, until the design is finalized.

D.

On Fri, Mar 10, 2017 at 11:55 AM, Denis Magda <[hidden email]> wrote:

> > Denis, I think we can allow this as well, but not as SPI since I want to
> > drop them and create 1 internal component to manage all
> interconnections. I
> > think this is enough since additions/changes will be rare and they will
> > result in changing internal code (including support of technologies you
> > mentioned). None of our users implemented discovery or communication on
> > their own, and I doubt if someone ever will. So, what is the purpose of
> > keeping SPIs?
>
> Yasha, I don’t insist to keep the SPIs but rather want to clarify that it
> will be feasible to provide a custom implementation of this new networking
> component if needed. Looks like this will be possible.
>
> > Dmitry, you are right - this seems to be a very complex change. However,
> we
> > can we move current implementations to internal packages, introduce
> > configuration properties/beans so we can release and continue development
> > in 2.0 without breaking compatibility.
>
> Let’s do this if there is a one who can take over the interface design and
> complete it by 2.0 release.
>
> —
> Denis
>
> > On Mar 10, 2017, at 7:39 AM, Yakov Zhdanov <[hidden email]> wrote:
> >
> > Dmitry, you are right - this seems to be a very complex change. However,
> we
> > can we move current implementations to internal packages, introduce
> > configuration properties/beans so we can release and continue development
> > in 2.0 without breaking compatibility.
> >
> > Denis, I think we can allow this as well, but not as SPI since I want to
> > drop them and create 1 internal component to manage all
> interconnections. I
> > think this is enough since additions/changes will be rare and they will
> > result in changing internal code (including support of technologies you
> > mentioned). None of our users implemented discovery or communication on
> > their own, and I doubt if someone ever will. So, what is the purpose of
> > keeping SPIs?
> >
> > --Yakov
>
>