[DISCUSSION] Modules organization in Ignite 3

[Alexey Goncharuk]

Igniters,

I want to tackle the topic of modules structure in Ignite 3. So far, the
modules in Ignite are mostly defined intuitively which leads to some
complications:

   - Ignite public API is separated from the rest of the code only by
   package name. This leads to private classes leaking to public API which is
   very hard to catch even during the review process (we missed a bunch of
   such leaks for new metrics API [1] and I remember this happening for almost
   every SPI)
   - Classes from 'internal' packages are considered to be 'free for grabs'
   in every place of the code. This leads to tight coupling and abstraction
   leakage in the code. An example of such a case - an often cast of
   WALPointer to FileWALPointer, so that the community decided to get rid of
   the WALPointer interface altogether [2]
   - Overall code complexity. Because of the lack of inter-module
   interaction rules, we are free to add new methods and callbacks to any
   class, which leads to duplicating entities and verbose interfaces. A good
   example of this is the clear duplication of methods in
   IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]

I think we need to work out some rules that will help us define and control
both Ignite public API and module internal API which still defines a clear
contract for other modules. Some ideas:

   - Perhaps we can move all user public classed and interfaces to an
   Ignite-API module which will have no dependencies on implementation
   modules. This will prevent private classes from leaking to the API module.
   - We need somehow define which classes from a module are exposed to
   other modules, and which classes are left for module-private usage. Maybe
   Java's jigsaw will help us here, but maybe we will be ok with just more
   strict java access modifiers usage :) The idea here is that a module should
   never touch a dependent module's private classes, ever. The exported
   classes and interfaces are still free to be modified between releases, as
   long as it is not a user public API.
   - A module should be logically complete, thus it may be beneficial if
   module name matches with the code package it provides (e.g. configuration
   -> org.apache.ignite.configuration, replication ->
   org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)

Any other principles/rules we can apply to make the code structure more
concise? Thoughts?

--AG

[1] https://issues.apache.org/jira/browse/IGNITE-12552
[2] https://issues.apache.org/jira/browse/IGNITE-13513
[3] https://issues.apache.org/jira/browse/IGNITE-13220

[Nikolay Izhikov-2]

Hello, Alexey.

Think we can extend our @IgniteExperimental annotation.

`@IgniteExperimental` - mark features that are truly experimental and can be completely removed in future releases.
`@NotRecommended` - mark features that widely adopted by the users but implemented wrong or have known issues that can’t be fixed.
`@NotStable` - mark features supported by community but API not stable and can be reworked in the next release.
`@Stable` - mark features that are completely OK and here to stay.

We should output notes about these annotations in the JavaDoc, also.
What do you think?

> 8 дек. 2020 г., в 12:49, Alexey Goncharuk <[hidden email]> написал(а):
>
> Igniters,
>
> I want to tackle the topic of modules structure in Ignite 3. So far, the
> modules in Ignite are mostly defined intuitively which leads to some
> complications:
>
>   - Ignite public API is separated from the rest of the code only by
>   package name. This leads to private classes leaking to public API which is
>   very hard to catch even during the review process (we missed a bunch of
>   such leaks for new metrics API [1] and I remember this happening for almost
>   every SPI)
>   - Classes from 'internal' packages are considered to be 'free for grabs'
>   in every place of the code. This leads to tight coupling and abstraction
>   leakage in the code. An example of such a case - an often cast of
>   WALPointer to FileWALPointer, so that the community decided to get rid of
>   the WALPointer interface altogether [2]
>   - Overall code complexity. Because of the lack of inter-module
>   interaction rules, we are free to add new methods and callbacks to any
>   class, which leads to duplicating entities and verbose interfaces. A good
>   example of this is the clear duplication of methods in
>   IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
>
> I think we need to work out some rules that will help us define and control
> both Ignite public API and module internal API which still defines a clear
> contract for other modules. Some ideas:
>
>   - Perhaps we can move all user public classed and interfaces to an
>   Ignite-API module which will have no dependencies on implementation
>   modules. This will prevent private classes from leaking to the API module.
>   - We need somehow define which classes from a module are exposed to
>   other modules, and which classes are left for module-private usage. Maybe
>   Java's jigsaw will help us here, but maybe we will be ok with just more
>   strict java access modifiers usage :) The idea here is that a module should
>   never touch a dependent module's private classes, ever. The exported
>   classes and interfaces are still free to be modified between releases, as
>   long as it is not a user public API.
>   - A module should be logically complete, thus it may be beneficial if
>   module name matches with the code package it provides (e.g. configuration
>   -> org.apache.ignite.configuration, replication ->
>   org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>
> Any other principles/rules we can apply to make the code structure more
> concise? Thoughts?
>
> --AG
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12552
> [2] https://issues.apache.org/jira/browse/IGNITE-13513
> [3] https://issues.apache.org/jira/browse/IGNITE-13220

[agura]

Definitely agree with Alexey. Separating API declaration from
implementation could really improve system design and avoid coupling.

About extending @IgniteExperimental annotation. It doesn't look good
to me. We should consider any API either experimental or stable. Third
option is deprecated API. Merging API which is known in advance that
it is unstable is a very bad idea.

On Tue, Dec 8, 2020 at 3:48 PM Nikolay Izhikov <[hidden email]> wrote:

>
> Hello, Alexey.
>
> Think we can extend our @IgniteExperimental annotation.
>
> `@IgniteExperimental` - mark features that are truly experimental and can be completely removed in future releases.
> `@NotRecommended` - mark features that widely adopted by the users but implemented wrong or have known issues that can’t be fixed.
> `@NotStable` - mark features supported by community but API not stable and can be reworked in the next release.
> `@Stable` - mark features that are completely OK and here to stay.
>
> We should output notes about these annotations in the JavaDoc, also.
> What do you think?
>
>
> > 8 дек. 2020 г., в 12:49, Alexey Goncharuk <[hidden email]> написал(а):
> >
> > Igniters,
> >
> > I want to tackle the topic of modules structure in Ignite 3. So far, the
> > modules in Ignite are mostly defined intuitively which leads to some
> > complications:
> >
> >   - Ignite public API is separated from the rest of the code only by
> >   package name. This leads to private classes leaking to public API which is
> >   very hard to catch even during the review process (we missed a bunch of
> >   such leaks for new metrics API [1] and I remember this happening for almost
> >   every SPI)
> >   - Classes from 'internal' packages are considered to be 'free for grabs'
> >   in every place of the code. This leads to tight coupling and abstraction
> >   leakage in the code. An example of such a case - an often cast of
> >   WALPointer to FileWALPointer, so that the community decided to get rid of
> >   the WALPointer interface altogether [2]
> >   - Overall code complexity. Because of the lack of inter-module
> >   interaction rules, we are free to add new methods and callbacks to any
> >   class, which leads to duplicating entities and verbose interfaces. A good
> >   example of this is the clear duplication of methods in
> >   IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
> >
> > I think we need to work out some rules that will help us define and control
> > both Ignite public API and module internal API which still defines a clear
> > contract for other modules. Some ideas:
> >
> >   - Perhaps we can move all user public classed and interfaces to an
> >   Ignite-API module which will have no dependencies on implementation
> >   modules. This will prevent private classes from leaking to the API module.
> >   - We need somehow define which classes from a module are exposed to
> >   other modules, and which classes are left for module-private usage. Maybe
> >   Java's jigsaw will help us here, but maybe we will be ok with just more
> >   strict java access modifiers usage :) The idea here is that a module should
> >   never touch a dependent module's private classes, ever. The exported
> >   classes and interfaces are still free to be modified between releases, as
> >   long as it is not a user public API.
> >   - A module should be logically complete, thus it may be beneficial if
> >   module name matches with the code package it provides (e.g. configuration
> >   -> org.apache.ignite.configuration, replication ->
> >   org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
> >
> > Any other principles/rules we can apply to make the code structure more
> > concise? Thoughts?
> >
> > --AG
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12552
> > [2] https://issues.apache.org/jira/browse/IGNITE-13513
> > [3] https://issues.apache.org/jira/browse/IGNITE-13220
>

[Kirill Tkalenko]

+1 for modules

08.12.2020, 16:02, "Andrey Gura" <[hidden email]>:

> Definitely agree with Alexey. Separating API declaration from
> implementation could really improve system design and avoid coupling.
>
> About extending @IgniteExperimental annotation. It doesn't look good
> to me. We should consider any API either experimental or stable. Third
> option is deprecated API. Merging API which is known in advance that
> it is unstable is a very bad idea.
>
> On Tue, Dec 8, 2020 at 3:48 PM Nikolay Izhikov <[hidden email]> wrote:
>>  Hello, Alexey.
>>
>>  Think we can extend our @IgniteExperimental annotation.
>>
>>  `@IgniteExperimental` - mark features that are truly experimental and can be completely removed in future releases.
>>  `@NotRecommended` - mark features that widely adopted by the users but implemented wrong or have known issues that can’t be fixed.
>>  `@NotStable` - mark features supported by community but API not stable and can be reworked in the next release.
>>  `@Stable` - mark features that are completely OK and here to stay.
>>
>>  We should output notes about these annotations in the JavaDoc, also.
>>  What do you think?
>>
>>  > 8 дек. 2020 г., в 12:49, Alexey Goncharuk <[hidden email]> написал(а):
>>  >
>>  > Igniters,
>>  >
>>  > I want to tackle the topic of modules structure in Ignite 3. So far, the
>>  > modules in Ignite are mostly defined intuitively which leads to some
>>  > complications:
>>  >
>>  > - Ignite public API is separated from the rest of the code only by
>>  > package name. This leads to private classes leaking to public API which is
>>  > very hard to catch even during the review process (we missed a bunch of
>>  > such leaks for new metrics API [1] and I remember this happening for almost
>>  > every SPI)
>>  > - Classes from 'internal' packages are considered to be 'free for grabs'
>>  > in every place of the code. This leads to tight coupling and abstraction
>>  > leakage in the code. An example of such a case - an often cast of
>>  > WALPointer to FileWALPointer, so that the community decided to get rid of
>>  > the WALPointer interface altogether [2]
>>  > - Overall code complexity. Because of the lack of inter-module
>>  > interaction rules, we are free to add new methods and callbacks to any
>>  > class, which leads to duplicating entities and verbose interfaces. A good
>>  > example of this is the clear duplication of methods in
>>  > IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
>>  >
>>  > I think we need to work out some rules that will help us define and control
>>  > both Ignite public API and module internal API which still defines a clear
>>  > contract for other modules. Some ideas:
>>  >
>>  > - Perhaps we can move all user public classed and interfaces to an
>>  > Ignite-API module which will have no dependencies on implementation
>>  > modules. This will prevent private classes from leaking to the API module.
>>  > - We need somehow define which classes from a module are exposed to
>>  > other modules, and which classes are left for module-private usage. Maybe
>>  > Java's jigsaw will help us here, but maybe we will be ok with just more
>>  > strict java access modifiers usage :) The idea here is that a module should
>>  > never touch a dependent module's private classes, ever. The exported
>>  > classes and interfaces are still free to be modified between releases, as
>>  > long as it is not a user public API.
>>  > - A module should be logically complete, thus it may be beneficial if
>>  > module name matches with the code package it provides (e.g. configuration
>>  > -> org.apache.ignite.configuration, replication ->
>>  > org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>>  >
>>  > Any other principles/rules we can apply to make the code structure more
>>  > concise? Thoughts?
>>  >
>>  > --AG
>>  >
>>  > [1] https://issues.apache.org/jira/browse/IGNITE-12552
>>  > [2] https://issues.apache.org/jira/browse/IGNITE-13513
>>  > [3] https://issues.apache.org/jira/browse/IGNITE-13220

[Valentin Kulichenko]

+1 to Alexey's ideas.

As for annotations, I actually can see the difference between Expirimental
and NonStable. The latter means that a feature works, but its API can still
be changed (in other words, no backwards compatibility guarantees). The
former is just something that is under development, can't be tried out, but
can't be relied on; potentially, something that can be removed completely.
I'm not sure if this can be applied to any of our current situations, but
as a general rule - makes sense to me.

I do not support adding the NotRecommended option though - such
functionality should be just deprecated.

-Val

On Tue, Dec 8, 2020 at 7:41 AM ткаленко кирилл <[hidden email]> wrote:

> +1 for modules
>
> 08.12.2020, 16:02, "Andrey Gura" <[hidden email]>:
> > Definitely agree with Alexey. Separating API declaration from
> > implementation could really improve system design and avoid coupling.
> >
> > About extending @IgniteExperimental annotation. It doesn't look good
> > to me. We should consider any API either experimental or stable. Third
> > option is deprecated API. Merging API which is known in advance that
> > it is unstable is a very bad idea.
> >
> > On Tue, Dec 8, 2020 at 3:48 PM Nikolay Izhikov <[hidden email]>
> wrote:
> >>  Hello, Alexey.
> >>
> >>  Think we can extend our @IgniteExperimental annotation.
> >>
> >>  `@IgniteExperimental` - mark features that are truly experimental and
> can be completely removed in future releases.
> >>  `@NotRecommended` - mark features that widely adopted by the users but
> implemented wrong or have known issues that can’t be fixed.
> >>  `@NotStable` - mark features supported by community but API not stable
> and can be reworked in the next release.
> >>  `@Stable` - mark features that are completely OK and here to stay.
> >>
> >>  We should output notes about these annotations in the JavaDoc, also.
> >>  What do you think?
> >>
> >>  > 8 дек. 2020 г., в 12:49, Alexey Goncharuk <
> [hidden email]> написал(а):
> >>  >
> >>  > Igniters,
> >>  >
> >>  > I want to tackle the topic of modules structure in Ignite 3. So far,
> the
> >>  > modules in Ignite are mostly defined intuitively which leads to some
> >>  > complications:
> >>  >
> >>  > - Ignite public API is separated from the rest of the code only by
> >>  > package name. This leads to private classes leaking to public API
> which is
> >>  > very hard to catch even during the review process (we missed a bunch
> of
> >>  > such leaks for new metrics API [1] and I remember this happening for
> almost
> >>  > every SPI)
> >>  > - Classes from 'internal' packages are considered to be 'free for
> grabs'
> >>  > in every place of the code. This leads to tight coupling and
> abstraction
> >>  > leakage in the code. An example of such a case - an often cast of
> >>  > WALPointer to FileWALPointer, so that the community decided to get
> rid of
> >>  > the WALPointer interface altogether [2]
> >>  > - Overall code complexity. Because of the lack of inter-module
> >>  > interaction rules, we are free to add new methods and callbacks to
> any
> >>  > class, which leads to duplicating entities and verbose interfaces. A
> good
> >>  > example of this is the clear duplication of methods in
> >>  > IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
> >>  >
> >>  > I think we need to work out some rules that will help us define and
> control
> >>  > both Ignite public API and module internal API which still defines a
> clear
> >>  > contract for other modules. Some ideas:
> >>  >
> >>  > - Perhaps we can move all user public classed and interfaces to an
> >>  > Ignite-API module which will have no dependencies on implementation
> >>  > modules. This will prevent private classes from leaking to the API
> module.
> >>  > - We need somehow define which classes from a module are exposed to
> >>  > other modules, and which classes are left for module-private usage.
> Maybe
> >>  > Java's jigsaw will help us here, but maybe we will be ok with just
> more
> >>  > strict java access modifiers usage :) The idea here is that a module
> should
> >>  > never touch a dependent module's private classes, ever. The exported
> >>  > classes and interfaces are still free to be modified between
> releases, as
> >>  > long as it is not a user public API.
> >>  > - A module should be logically complete, thus it may be beneficial if
> >>  > module name matches with the code package it provides (e.g.
> configuration
> >>  > -> org.apache.ignite.configuration, replication ->
> >>  > org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
> >>  >
> >>  > Any other principles/rules we can apply to make the code structure
> more
> >>  > concise? Thoughts?
> >>  >
> >>  > --AG
> >>  >
> >>  > [1] https://issues.apache.org/jira/browse/IGNITE-12552
> >>  > [2] https://issues.apache.org/jira/browse/IGNITE-13513
> >>  > [3] https://issues.apache.org/jira/browse/IGNITE-13220
>

[Zhenya Stanilovsky]

Hello Nikolay, if i find out introduced features structure in some project, i would prefer to choose different one )
 

> 
>> 
>>>Hello, Alexey.
>>>
>>>Think we can extend our @IgniteExperimental annotation.
>>>
>>>`@IgniteExperimental` - mark features that are truly experimental and can be completely removed in future releases.
>>>`@NotRecommended` - mark features that widely adopted by the users but implemented wrong or have known issues that can’t be fixed.
>>>`@NotStable` - mark features supported by community but API not stable and can be reworked in the next release.
>>>`@Stable` - mark features that are completely OK and here to stay.
>>>
>>>We should output notes about these annotations in the JavaDoc, also.
>>>What do you think?
>>>
>>> 
>>>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk < [hidden email] > написал(а):
>>>>
>>>> Igniters,
>>>>
>>>> I want to tackle the topic of modules structure in Ignite 3. So far, the
>>>> modules in Ignite are mostly defined intuitively which leads to some
>>>> complications:
>>>>
>>>> - Ignite public API is separated from the rest of the code only by
>>>> package name. This leads to private classes leaking to public API which is
>>>> very hard to catch even during the review process (we missed a bunch of
>>>> such leaks for new metrics API [1] and I remember this happening for almost
>>>> every SPI)
>>>> - Classes from 'internal' packages are considered to be 'free for grabs'
>>>> in every place of the code. This leads to tight coupling and abstraction
>>>> leakage in the code. An example of such a case - an often cast of
>>>> WALPointer to FileWALPointer, so that the community decided to get rid of
>>>> the WALPointer interface altogether [2]
>>>> - Overall code complexity. Because of the lack of inter-module
>>>> interaction rules, we are free to add new methods and callbacks to any
>>>> class, which leads to duplicating entities and verbose interfaces. A good
>>>> example of this is the clear duplication of methods in
>>>> IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
>>>>
>>>> I think we need to work out some rules that will help us define and control
>>>> both Ignite public API and module internal API which still defines a clear
>>>> contract for other modules. Some ideas:
>>>>
>>>> - Perhaps we can move all user public classed and interfaces to an
>>>> Ignite-API module which will have no dependencies on implementation
>>>> modules. This will prevent private classes from leaking to the API module.
>>>> - We need somehow define which classes from a module are exposed to
>>>> other modules, and which classes are left for module-private usage. Maybe
>>>> Java's jigsaw will help us here, but maybe we will be ok with just more
>>>> strict java access modifiers usage :) The idea here is that a module should
>>>> never touch a dependent module's private classes, ever. The exported
>>>> classes and interfaces are still free to be modified between releases, as
>>>> long as it is not a user public API.
>>>> - A module should be logically complete, thus it may be beneficial if
>>>> module name matches with the code package it provides (e.g. configuration
>>>> -> org.apache.ignite.configuration, replication ->
>>>> org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>>>>
>>>> Any other principles/rules we can apply to make the code structure more
>>>> concise? Thoughts?
>>>>
>>>> --AG
>>>>
>>>> [1]  https://issues.apache.org/jira/browse/IGNITE-12552
>>>> [2]  https://issues.apache.org/jira/browse/IGNITE-13513
>>>> [3]  https://issues.apache.org/jira/browse/IGNITE-13220 
>> 
>> 
>> 
>> 

[Ivan Bessonov]

Conversation shifted into an unintended direction, but I agree.

I think that if API can (or will) be changed then it should be deprecated.
For that
we can introduce @IgniteDeprecated that will contain Ignite version when
API is planned to be removed. Otherwise it's either stable or experimental.
Having officially "unstable" features doesn't sound good for product
reputation.

As for the modularization - I'm all for this idea. If we don't force
ourselves to
organize code properly then we'll end up with the same problems as we have
in the current code base. And this way there's a hope of having good tests
that can be completed in minutes, not hours. At least new ones.

BTW, did we have any discussions about dependency injection and all this
stuff?
Seems like a related topic to me.

ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky <[hidden email]>:

>
>
> Hello Nikolay, if i find out introduced features structure in some
> project, i would prefer to choose different one )
>
> >
> >>
> >>>Hello, Alexey.
> >>>
> >>>Think we can extend our @IgniteExperimental annotation.
> >>>
> >>>`@IgniteExperimental` - mark features that are truly experimental and
> can be completely removed in future releases.
> >>>`@NotRecommended` - mark features that widely adopted by the users but
> implemented wrong or have known issues that can’t be fixed.
> >>>`@NotStable` - mark features supported by community but API not stable
> and can be reworked in the next release.
> >>>`@Stable` - mark features that are completely OK and here to stay.
> >>>
> >>>We should output notes about these annotations in the JavaDoc, also.
> >>>What do you think?
> >>>
> >>>
> >>>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk <
> [hidden email] > написал(а):
> >>>>
> >>>> Igniters,
> >>>>
> >>>> I want to tackle the topic of modules structure in Ignite 3. So far,
> the
> >>>> modules in Ignite are mostly defined intuitively which leads to some
> >>>> complications:
> >>>>
> >>>> - Ignite public API is separated from the rest of the code only by
> >>>> package name. This leads to private classes leaking to public API
> which is
> >>>> very hard to catch even during the review process (we missed a bunch
> of
> >>>> such leaks for new metrics API [1] and I remember this happening for
> almost
> >>>> every SPI)
> >>>> - Classes from 'internal' packages are considered to be 'free for
> grabs'
> >>>> in every place of the code. This leads to tight coupling and
> abstraction
> >>>> leakage in the code. An example of such a case - an often cast of
> >>>> WALPointer to FileWALPointer, so that the community decided to get
> rid of
> >>>> the WALPointer interface altogether [2]
> >>>> - Overall code complexity. Because of the lack of inter-module
> >>>> interaction rules, we are free to add new methods and callbacks to any
> >>>> class, which leads to duplicating entities and verbose interfaces. A
> good
> >>>> example of this is the clear duplication of methods in
> >>>> IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
> >>>>
> >>>> I think we need to work out some rules that will help us define and
> control
> >>>> both Ignite public API and module internal API which still defines a
> clear
> >>>> contract for other modules. Some ideas:
> >>>>
> >>>> - Perhaps we can move all user public classed and interfaces to an
> >>>> Ignite-API module which will have no dependencies on implementation
> >>>> modules. This will prevent private classes from leaking to the API
> module.
> >>>> - We need somehow define which classes from a module are exposed to
> >>>> other modules, and which classes are left for module-private usage.
> Maybe
> >>>> Java's jigsaw will help us here, but maybe we will be ok with just
> more
> >>>> strict java access modifiers usage :) The idea here is that a module
> should
> >>>> never touch a dependent module's private classes, ever. The exported
> >>>> classes and interfaces are still free to be modified between
> releases, as
> >>>> long as it is not a user public API.
> >>>> - A module should be logically complete, thus it may be beneficial if
> >>>> module name matches with the code package it provides (e.g.
> configuration
> >>>> -> org.apache.ignite.configuration, replication ->
> >>>> org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
> >>>>
> >>>> Any other principles/rules we can apply to make the code structure
> more
> >>>> concise? Thoughts?
> >>>>
> >>>> --AG
> >>>>
> >>>> [1]  https://issues.apache.org/jira/browse/IGNITE-12552
> >>>> [2]  https://issues.apache.org/jira/browse/IGNITE-13513
> >>>> [3]  https://issues.apache.org/jira/browse/IGNITE-13220
> >>
> >>
> >>
> >>

--
Sincerely yours,
Ivan Bessonov

[Nikolay Izhikov-2]

Hello, Zhenya, Ivan.

> Hello Nikolay, if i find out introduced features structure in some project, i would prefer to choose different one )

Many, of the real world users disagree with you.
Please, take a look at some examples from widely used projects:

Kafka - https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
        - Stable, Evolving, Unstable

Spark - https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
        - AlphaComponent, DeveloperApi, Evolving, Experimental, Private, Stable, Unstable

> Having officially "unstable" features doesn't sound good for product reputation.

Can’t agree with you.

Forcing ourselves to make perfect API from the first try we just put too much pressure on every decision.
Every developer making mistakes.
The product is evolving and the API too - it’s totally OK.

For every new feature time required to be adopted and used in real-world production.
I believe, slight API changes is totally fine for early adopters.
Moreover, I think, that we should warn our users that some feature is very fresh and can have issues.

So, Why Kafka and Spark is good enough to have unstable API and Ignite not? :)

> 9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]> написал(а):
>
> Conversation shifted into an unintended direction, but I agree.
>
> I think that if API can (or will) be changed then it should be deprecated.
> For that
> we can introduce @IgniteDeprecated that will contain Ignite version when
> API is planned to be removed. Otherwise it's either stable or experimental.
> Having officially "unstable" features doesn't sound good for product
> reputation.
>
> As for the modularization - I'm all for this idea. If we don't force
> ourselves to
> organize code properly then we'll end up with the same problems as we have
> in the current code base. And this way there's a hope of having good tests
> that can be completed in minutes, not hours. At least new ones.
>
> BTW, did we have any discussions about dependency injection and all this
> stuff?
> Seems like a related topic to me.
>
> ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky <[hidden email]>:
>
>>
>>
>> Hello Nikolay, if i find out introduced features structure in some
>> project, i would prefer to choose different one )
>>
>>>
>>>>
>>>>> Hello, Alexey.
>>>>>
>>>>> Think we can extend our @IgniteExperimental annotation.
>>>>>
>>>>> `@IgniteExperimental` - mark features that are truly experimental and
>> can be completely removed in future releases.
>>>>> `@NotRecommended` - mark features that widely adopted by the users but
>> implemented wrong or have known issues that can’t be fixed.
>>>>> `@NotStable` - mark features supported by community but API not stable
>> and can be reworked in the next release.
>>>>> `@Stable` - mark features that are completely OK and here to stay.
>>>>>
>>>>> We should output notes about these annotations in the JavaDoc, also.
>>>>> What do you think?
>>>>>
>>>>>
>>>>>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk <
>> [hidden email] > написал(а):
>>>>>>
>>>>>> Igniters,
>>>>>>
>>>>>> I want to tackle the topic of modules structure in Ignite 3. So far,
>> the
>>>>>> modules in Ignite are mostly defined intuitively which leads to some
>>>>>> complications:
>>>>>>
>>>>>> - Ignite public API is separated from the rest of the code only by
>>>>>> package name. This leads to private classes leaking to public API
>> which is
>>>>>> very hard to catch even during the review process (we missed a bunch
>> of
>>>>>> such leaks for new metrics API [1] and I remember this happening for
>> almost
>>>>>> every SPI)
>>>>>> - Classes from 'internal' packages are considered to be 'free for
>> grabs'
>>>>>> in every place of the code. This leads to tight coupling and
>> abstraction
>>>>>> leakage in the code. An example of such a case - an often cast of
>>>>>> WALPointer to FileWALPointer, so that the community decided to get
>> rid of
>>>>>> the WALPointer interface altogether [2]
>>>>>> - Overall code complexity. Because of the lack of inter-module
>>>>>> interaction rules, we are free to add new methods and callbacks to any
>>>>>> class, which leads to duplicating entities and verbose interfaces. A
>> good
>>>>>> example of this is the clear duplication of methods in
>>>>>> IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
>>>>>>
>>>>>> I think we need to work out some rules that will help us define and
>> control
>>>>>> both Ignite public API and module internal API which still defines a
>> clear
>>>>>> contract for other modules. Some ideas:
>>>>>>
>>>>>> - Perhaps we can move all user public classed and interfaces to an
>>>>>> Ignite-API module which will have no dependencies on implementation
>>>>>> modules. This will prevent private classes from leaking to the API
>> module.
>>>>>> - We need somehow define which classes from a module are exposed to
>>>>>> other modules, and which classes are left for module-private usage.
>> Maybe
>>>>>> Java's jigsaw will help us here, but maybe we will be ok with just
>> more
>>>>>> strict java access modifiers usage :) The idea here is that a module
>> should
>>>>>> never touch a dependent module's private classes, ever. The exported
>>>>>> classes and interfaces are still free to be modified between
>> releases, as
>>>>>> long as it is not a user public API.
>>>>>> - A module should be logically complete, thus it may be beneficial if
>>>>>> module name matches with the code package it provides (e.g.
>> configuration
>>>>>> -> org.apache.ignite.configuration, replication ->
>>>>>> org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>>>>>>
>>>>>> Any other principles/rules we can apply to make the code structure
>> more
>>>>>> concise? Thoughts?
>>>>>>
>>>>>> --AG
>>>>>>
>>>>>> [1]  https://issues.apache.org/jira/browse/IGNITE-12552
>>>>>> [2]  https://issues.apache.org/jira/browse/IGNITE-13513
>>>>>> [3]  https://issues.apache.org/jira/browse/IGNITE-13220
>>>>
>>>>
>>>>
>>>>
>
>
>
> --
> Sincerely yours,
> Ivan Bessonov

[Anton Kalashnikov]

Hello,

I totally agree that we should start to think about the module organization.
I suggest making the new confluence page where we define new rules on how to develop modules.
In my opinion, at least, the following topics should be covered there(it makes sense to discuss every topic separately, not here):
* In which cases new modules are required
* The naming of modules, packages
* Class dependency management ( inversion of control, no more context)
* Test organization ( module contains only unit, module tests. All integration tests are in the extra module)
* Feature/Module lifecycle - experimental, stable, unstable, deprecated


Let's get back to the original topic. I agree with the package naming rule:
if the module name is configuration the package name should be org.apache.ignite.configuration and only after that any other subpackages.
Also, I don't sure that we need ignite- prefix in the module name because it doesn't have any extra information:

  <groupId>org.apache.ignite</groupId>
  <artifactId>ignite-configuration</artifactId>

we don't lose anything if convert it to

  <groupId>org.apache.ignite</groupId>
  <artifactId>configuration</artifactId>


I also hope that jigsaw can help us somehow with class visibility.
But if not we can take agreement that for example 'internal' package - shouldn't be touched outside of the module -
of course, using the proper class access level is also the solution(where it is possible)
org.apache.ignite.configuration.Configurator // it has access to the internal package but it shouldn't return any class from the internal package only from the public one.
org.apache.ignite.configuration.DynamicProperty //interface
org.apache.ignite.configuration.internal.properties.IntDynamicProperty


User API makes sense only for one end module - ignite(or ignite-core) which depends on all other modules
and doing some integration(adapters) and provide final API for the user.
So I agree that separated module Ignite-API with zero dependencies will be a good solution.

configuration module:
  Configurator.baseline().enabled() -> DynamicProperties

ignite-api module:
  BaselineConfiguration.enabled() -> boolean //interface

ignite module:
  BaselineConfigurationImpl implements BaselineConfiguration{
    Configurator configurator;
    public boolean enabled(){
       return configurator.baseline().enabled().value();
    }
  }

So maybe my example is not so good. But I want to show that end-user API will be defined only in ignite-api
and you need to adapt it in ignite module which leads to some overhead(especially in my example)
 but it makes development pretty manageable/predictable -
you can easily implement a new module without any worries that user starts to use it.
It will be available only after making changes in ignite-api.
The major advantage here is the small size of ignite-api which allows to carefully review every change
which allows keeping ignite API in better quality(I hope at least)

Nikolay, maybe is it better to discuss your question on a separate topic? Because looks like it is a pretty discussable topic.

-- 
Best regards,
Anton Kalashnikov



09.12.2020, 10:31, "Nikolay Izhikov" <[hidden email]>:

> Hello, Zhenya, Ivan.
>
>>  Hello Nikolay, if i find out introduced features structure in some project, i would prefer to choose different one )
>
> Many, of the real world users disagree with you.
> Please, take a look at some examples from widely used projects:
>
> Kafka - https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
>         - Stable, Evolving, Unstable
>
> Spark - https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
>         - AlphaComponent, DeveloperApi, Evolving, Experimental, Private, Stable, Unstable
>
>>  Having officially "unstable" features doesn't sound good for product reputation.
>
> Can’t agree with you.
>
> Forcing ourselves to make perfect API from the first try we just put too much pressure on every decision.
> Every developer making mistakes.
> The product is evolving and the API too - it’s totally OK.
>
> For every new feature time required to be adopted and used in real-world production.
> I believe, slight API changes is totally fine for early adopters.
> Moreover, I think, that we should warn our users that some feature is very fresh and can have issues.
>
> So, Why Kafka and Spark is good enough to have unstable API and Ignite not? :)
>
>>  9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]> написал(а):
>>
>>  Conversation shifted into an unintended direction, but I agree.
>>
>>  I think that if API can (or will) be changed then it should be deprecated.
>>  For that
>>  we can introduce @IgniteDeprecated that will contain Ignite version when
>>  API is planned to be removed. Otherwise it's either stable or experimental.
>>  Having officially "unstable" features doesn't sound good for product
>>  reputation.
>>
>>  As for the modularization - I'm all for this idea. If we don't force
>>  ourselves to
>>  organize code properly then we'll end up with the same problems as we have
>>  in the current code base. And this way there's a hope of having good tests
>>  that can be completed in minutes, not hours. At least new ones.
>>
>>  BTW, did we have any discussions about dependency injection and all this
>>  stuff?
>>  Seems like a related topic to me.
>>
>>  ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky <[hidden email]>:
>>
>>>  Hello Nikolay, if i find out introduced features structure in some
>>>  project, i would prefer to choose different one )
>>>
>>>>>>  Hello, Alexey.
>>>>>>
>>>>>>  Think we can extend our @IgniteExperimental annotation.
>>>>>>
>>>>>>  `@IgniteExperimental` - mark features that are truly experimental and
>>>  can be completely removed in future releases.
>>>>>>  `@NotRecommended` - mark features that widely adopted by the users but
>>>  implemented wrong or have known issues that can’t be fixed.
>>>>>>  `@NotStable` - mark features supported by community but API not stable
>>>  and can be reworked in the next release.
>>>>>>  `@Stable` - mark features that are completely OK and here to stay.
>>>>>>
>>>>>>  We should output notes about these annotations in the JavaDoc, also.
>>>>>>  What do you think?
>>>>>>
>>>>>>>  8 дек. 2020 г., в 12:49, Alexey Goncharuk <
>>>  [hidden email] > написал(а):
>>>>>>>  Igniters,
>>>>>>>
>>>>>>>  I want to tackle the topic of modules structure in Ignite 3. So far,
>>>  the
>>>>>>>  modules in Ignite are mostly defined intuitively which leads to some
>>>>>>>  complications:
>>>>>>>
>>>>>>>  - Ignite public API is separated from the rest of the code only by
>>>>>>>  package name. This leads to private classes leaking to public API
>>>  which is
>>>>>>>  very hard to catch even during the review process (we missed a bunch
>>>  of
>>>>>>>  such leaks for new metrics API [1] and I remember this happening for
>>>  almost
>>>>>>>  every SPI)
>>>>>>>  - Classes from 'internal' packages are considered to be 'free for
>>>  grabs'
>>>>>>>  in every place of the code. This leads to tight coupling and
>>>  abstraction
>>>>>>>  leakage in the code. An example of such a case - an often cast of
>>>>>>>  WALPointer to FileWALPointer, so that the community decided to get
>>>  rid of
>>>>>>>  the WALPointer interface altogether [2]
>>>>>>>  - Overall code complexity. Because of the lack of inter-module
>>>>>>>  interaction rules, we are free to add new methods and callbacks to any
>>>>>>>  class, which leads to duplicating entities and verbose interfaces. A
>>>  good
>>>>>>>  example of this is the clear duplication of methods in
>>>>>>>  IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
>>>>>>>
>>>>>>>  I think we need to work out some rules that will help us define and
>>>  control
>>>>>>>  both Ignite public API and module internal API which still defines a
>>>  clear
>>>>>>>  contract for other modules. Some ideas:
>>>>>>>
>>>>>>>  - Perhaps we can move all user public classed and interfaces to an
>>>>>>>  Ignite-API module which will have no dependencies on implementation
>>>>>>>  modules. This will prevent private classes from leaking to the API
>>>  module.
>>>>>>>  - We need somehow define which classes from a module are exposed to
>>>>>>>  other modules, and which classes are left for module-private usage.
>>>  Maybe
>>>>>>>  Java's jigsaw will help us here, but maybe we will be ok with just
>>>  more
>>>>>>>  strict java access modifiers usage :) The idea here is that a module
>>>  should
>>>>>>>  never touch a dependent module's private classes, ever. The exported
>>>>>>>  classes and interfaces are still free to be modified between
>>>  releases, as
>>>>>>>  long as it is not a user public API.
>>>>>>>  - A module should be logically complete, thus it may be beneficial if
>>>>>>>  module name matches with the code package it provides (e.g.
>>>  configuration
>>>>>>>  -> org.apache.ignite.configuration, replication ->
>>>>>>>  org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>>>>>>>
>>>>>>>  Any other principles/rules we can apply to make the code structure
>>>  more
>>>>>>>  concise? Thoughts?
>>>>>>>
>>>>>>>  --AG
>>>>>>>
>>>>>>>  [1] https://issues.apache.org/jira/browse/IGNITE-12552
>>>>>>>  [2] https://issues.apache.org/jira/browse/IGNITE-13513
>>>>>>>  [3] https://issues.apache.org/jira/browse/IGNITE-13220
>>
>>  --
>>  Sincerely yours,
>>  Ivan Bessonov

[Nikolay Izhikov-2]

> Nikolay, maybe is it better to discuss your question on a separate topic? Because looks like it is a pretty discussable topic.

Sorry for interrupting original discussion.
For me it was a overlapping topic in the beginning.
Agree that we  we should have separate thread.

> 9 дек. 2020 г., в 16:15, Anton Kalashnikov <[hidden email]> написал(а):
>
> Hello,
>
> I totally agree that we should start to think about the module organization.
> I suggest making the new confluence page where we define new rules on how to develop modules.
> In my opinion, at least, the following topics should be covered there(it makes sense to discuss every topic separately, not here):
> * In which cases new modules are required
> * The naming of modules, packages
> * Class dependency management ( inversion of control, no more context)
> * Test organization ( module contains only unit, module tests. All integration tests are in the extra module)
> * Feature/Module lifecycle - experimental, stable, unstable, deprecated
>
>
> Let's get back to the original topic. I agree with the package naming rule:
> if the module name is configuration the package name should be org.apache.ignite.configuration and only after that any other subpackages.
> Also, I don't sure that we need ignite- prefix in the module name because it doesn't have any extra information:
>
>  <groupId>org.apache.ignite</groupId>
>  <artifactId>ignite-configuration</artifactId>
>
> we don't lose anything if convert it to
>
>  <groupId>org.apache.ignite</groupId>
>  <artifactId>configuration</artifactId>
>
>
> I also hope that jigsaw can help us somehow with class visibility.
> But if not we can take agreement that for example 'internal' package - shouldn't be touched outside of the module -
> of course, using the proper class access level is also the solution(where it is possible)
> org.apache.ignite.configuration.Configurator // it has access to the internal package but it shouldn't return any class from the internal package only from the public one.
> org.apache.ignite.configuration.DynamicProperty //interface
> org.apache.ignite.configuration.internal.properties.IntDynamicProperty
>
>
> User API makes sense only for one end module - ignite(or ignite-core) which depends on all other modules
> and doing some integration(adapters) and provide final API for the user.
> So I agree that separated module Ignite-API with zero dependencies will be a good solution.
>
> configuration module:
>  Configurator.baseline().enabled() -> DynamicProperties
>
> ignite-api module:
>  BaselineConfiguration.enabled() -> boolean //interface
>
> ignite module:
>  BaselineConfigurationImpl implements BaselineConfiguration{
>     Configurator configurator;
>     public boolean enabled(){
>        return configurator.baseline().enabled().value();
>     }
>  }
>
> So maybe my example is not so good. But I want to show that end-user API will be defined only in ignite-api
> and you need to adapt it in ignite module which leads to some overhead(especially in my example)
> but it makes development pretty manageable/predictable -
> you can easily implement a new module without any worries that user starts to use it.
> It will be available only after making changes in ignite-api.
> The major advantage here is the small size of ignite-api which allows to carefully review every change
> which allows keeping ignite API in better quality(I hope at least)
>
> Nikolay, maybe is it better to discuss your question on a separate topic? Because looks like it is a pretty discussable topic.
>
> --
> Best regards,
> Anton Kalashnikov
>
>
>
> 09.12.2020, 10:31, "Nikolay Izhikov" <[hidden email]>:
>> Hello, Zhenya, Ivan.
>>
>>>  Hello Nikolay, if i find out introduced features structure in some project, i would prefer to choose different one )
>>
>> Many, of the real world users disagree with you.
>> Please, take a look at some examples from widely used projects:
>>
>> Kafka - https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
>>         - Stable, Evolving, Unstable
>>
>> Spark - https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
>>         - AlphaComponent, DeveloperApi, Evolving, Experimental, Private, Stable, Unstable
>>
>>>  Having officially "unstable" features doesn't sound good for product reputation.
>>
>> Can’t agree with you.
>>
>> Forcing ourselves to make perfect API from the first try we just put too much pressure on every decision.
>> Every developer making mistakes.
>> The product is evolving and the API too - it’s totally OK.
>>
>> For every new feature time required to be adopted and used in real-world production.
>> I believe, slight API changes is totally fine for early adopters.
>> Moreover, I think, that we should warn our users that some feature is very fresh and can have issues.
>>
>> So, Why Kafka and Spark is good enough to have unstable API and Ignite not? :)
>>
>>>  9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]> написал(а):
>>>
>>>  Conversation shifted into an unintended direction, but I agree.
>>>
>>>  I think that if API can (or will) be changed then it should be deprecated.
>>>  For that
>>>  we can introduce @IgniteDeprecated that will contain Ignite version when
>>>  API is planned to be removed. Otherwise it's either stable or experimental.
>>>  Having officially "unstable" features doesn't sound good for product
>>>  reputation.
>>>
>>>  As for the modularization - I'm all for this idea. If we don't force
>>>  ourselves to
>>>  organize code properly then we'll end up with the same problems as we have
>>>  in the current code base. And this way there's a hope of having good tests
>>>  that can be completed in minutes, not hours. At least new ones.
>>>
>>>  BTW, did we have any discussions about dependency injection and all this
>>>  stuff?
>>>  Seems like a related topic to me.
>>>
>>>  ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky <[hidden email]>:
>>>
>>>>  Hello Nikolay, if i find out introduced features structure in some
>>>>  project, i would prefer to choose different one )
>>>>
>>>>>>>  Hello, Alexey.
>>>>>>>
>>>>>>>  Think we can extend our @IgniteExperimental annotation.
>>>>>>>
>>>>>>>  `@IgniteExperimental` - mark features that are truly experimental and
>>>>  can be completely removed in future releases.
>>>>>>>  `@NotRecommended` - mark features that widely adopted by the users but
>>>>  implemented wrong or have known issues that can’t be fixed.
>>>>>>>  `@NotStable` - mark features supported by community but API not stable
>>>>  and can be reworked in the next release.
>>>>>>>  `@Stable` - mark features that are completely OK and here to stay.
>>>>>>>
>>>>>>>  We should output notes about these annotations in the JavaDoc, also.
>>>>>>>  What do you think?
>>>>>>>
>>>>>>>>  8 дек. 2020 г., в 12:49, Alexey Goncharuk <
>>>>  [hidden email] > написал(а):
>>>>>>>>  Igniters,
>>>>>>>>
>>>>>>>>  I want to tackle the topic of modules structure in Ignite 3. So far,
>>>>  the
>>>>>>>>  modules in Ignite are mostly defined intuitively which leads to some
>>>>>>>>  complications:
>>>>>>>>
>>>>>>>>  - Ignite public API is separated from the rest of the code only by
>>>>>>>>  package name. This leads to private classes leaking to public API
>>>>  which is
>>>>>>>>  very hard to catch even during the review process (we missed a bunch
>>>>  of
>>>>>>>>  such leaks for new metrics API [1] and I remember this happening for
>>>>  almost
>>>>>>>>  every SPI)
>>>>>>>>  - Classes from 'internal' packages are considered to be 'free for
>>>>  grabs'
>>>>>>>>  in every place of the code. This leads to tight coupling and
>>>>  abstraction
>>>>>>>>  leakage in the code. An example of such a case - an often cast of
>>>>>>>>  WALPointer to FileWALPointer, so that the community decided to get
>>>>  rid of
>>>>>>>>  the WALPointer interface altogether [2]
>>>>>>>>  - Overall code complexity. Because of the lack of inter-module
>>>>>>>>  interaction rules, we are free to add new methods and callbacks to any
>>>>>>>>  class, which leads to duplicating entities and verbose interfaces. A
>>>>  good
>>>>>>>>  example of this is the clear duplication of methods in
>>>>>>>>  IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3]
>>>>>>>>
>>>>>>>>  I think we need to work out some rules that will help us define and
>>>>  control
>>>>>>>>  both Ignite public API and module internal API which still defines a
>>>>  clear
>>>>>>>>  contract for other modules. Some ideas:
>>>>>>>>
>>>>>>>>  - Perhaps we can move all user public classed and interfaces to an
>>>>>>>>  Ignite-API module which will have no dependencies on implementation
>>>>>>>>  modules. This will prevent private classes from leaking to the API
>>>>  module.
>>>>>>>>  - We need somehow define which classes from a module are exposed to
>>>>>>>>  other modules, and which classes are left for module-private usage.
>>>>  Maybe
>>>>>>>>  Java's jigsaw will help us here, but maybe we will be ok with just
>>>>  more
>>>>>>>>  strict java access modifiers usage :) The idea here is that a module
>>>>  should
>>>>>>>>  never touch a dependent module's private classes, ever. The exported
>>>>>>>>  classes and interfaces are still free to be modified between
>>>>  releases, as
>>>>>>>>  long as it is not a user public API.
>>>>>>>>  - A module should be logically complete, thus it may be beneficial if
>>>>>>>>  module name matches with the code package it provides (e.g.
>>>>  configuration
>>>>>>>>  -> org.apache.ignite.configuration, replication ->
>>>>>>>>  org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>>>>>>>>
>>>>>>>>  Any other principles/rules we can apply to make the code structure
>>>>  more
>>>>>>>>  concise? Thoughts?
>>>>>>>>
>>>>>>>>  --AG
>>>>>>>>
>>>>>>>>  [1] https://issues.apache.org/jira/browse/IGNITE-12552
>>>>>>>>  [2] https://issues.apache.org/jira/browse/IGNITE-13513
>>>>>>>>  [3] https://issues.apache.org/jira/browse/IGNITE-13220
>>>
>>>  --
>>>  Sincerely yours,
>>>  Ivan Bessonov

[Ilya Kasnacheev]

Hello!

When you do mvn dependencies:copy, you will now end up with
"ignite-core-X.Y.Z.jar", "ignite-indexing-X.Y.Z.jar"
But if we remove "ignite" from artifact name, user will end up with cryptic
"configuration.jar", "raft.jar", etc, etc.

Remember that group name is discarded in file names. Therefore, every
project out there keeps stuff like "spring-" or "hibernate-" in artifact
name.

I also hope that our users will not end up with zillion JARs.

Regards,
--
Ilya Kasnacheev


ср, 9 дек. 2020 г. в 16:16, Anton Kalashnikov <[hidden email]>:

> Hello,
>
> I totally agree that we should start to think about the module
> organization.
> I suggest making the new confluence page where we define new rules on how
> to develop modules.
> In my opinion, at least, the following topics should be covered there(it
> makes sense to discuss every topic separately, not here):
> * In which cases new modules are required
> * The naming of modules, packages
> * Class dependency management ( inversion of control, no more context)
> * Test organization ( module contains only unit, module tests. All
> integration tests are in the extra module)
> * Feature/Module lifecycle - experimental, stable, unstable, deprecated
>
>
> Let's get back to the original topic. I agree with the package naming
> rule:
> if the module name is configuration the package name should be
> org.apache.ignite.configuration and only after that any other subpackages.
> Also, I don't sure that we need ignite- prefix in the module name because
> it doesn't have any extra information:
>
>   <groupId>org.apache.ignite</groupId>
>   <artifactId>ignite-configuration</artifactId>
>
> we don't lose anything if convert it to
>
>   <groupId>org.apache.ignite</groupId>
>   <artifactId>configuration</artifactId>
>
>
> I also hope that jigsaw can help us somehow with class visibility.
> But if not we can take agreement that for example 'internal' package -
> shouldn't be touched outside of the module -
> of course, using the proper class access level is also the solution(where
> it is possible)
> org.apache.ignite.configuration.Configurator // it has access to the
> internal package but it shouldn't return any class from the internal
> package only from the public one.
> org.apache.ignite.configuration.DynamicProperty //interface
> org.apache.ignite.configuration.internal.properties.IntDynamicProperty
>
>
> User API makes sense only for one end module - ignite(or ignite-core)
> which depends on all other modules
> and doing some integration(adapters) and provide final API for the user.
> So I agree that separated module Ignite-API with zero dependencies will be
> a good solution.
>
> configuration module:
>   Configurator.baseline().enabled() -> DynamicProperties
>
> ignite-api module:
>   BaselineConfiguration.enabled() -> boolean //interface
>
> ignite module:
>   BaselineConfigurationImpl implements BaselineConfiguration{
>     Configurator configurator;
>     public boolean enabled(){
>        return configurator.baseline().enabled().value();
>     }
>   }
>
> So maybe my example is not so good. But I want to show that end-user API
> will be defined only in ignite-api
> and you need to adapt it in ignite module which leads to some
> overhead(especially in my example)
>  but it makes development pretty manageable/predictable -
> you can easily implement a new module without any worries that user starts
> to use it.
> It will be available only after making changes in ignite-api.
> The major advantage here is the small size of ignite-api which allows to
> carefully review every change
> which allows keeping ignite API in better quality(I hope at least)
>
> Nikolay, maybe is it better to discuss your question on a separate topic?
> Because looks like it is a pretty discussable topic.
>
> --
> Best regards,
> Anton Kalashnikov
>
>
>
> 09.12.2020, 10:31, "Nikolay Izhikov" <[hidden email]>:
> > Hello, Zhenya, Ivan.
> >
> >>  Hello Nikolay, if i find out introduced features structure in some
> project, i would prefer to choose different one )
> >
> > Many, of the real world users disagree with you.
> > Please, take a look at some examples from widely used projects:
> >
> > Kafka -
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
> >         - Stable, Evolving, Unstable
> >
> > Spark -
> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
> >         - AlphaComponent, DeveloperApi, Evolving, Experimental, Private,
> Stable, Unstable
> >
> >>  Having officially "unstable" features doesn't sound good for product
> reputation.
> >
> > Can’t agree with you.
> >
> > Forcing ourselves to make perfect API from the first try we just put too
> much pressure on every decision.
> > Every developer making mistakes.
> > The product is evolving and the API too - it’s totally OK.
> >
> > For every new feature time required to be adopted and used in real-world
> production.
> > I believe, slight API changes is totally fine for early adopters.
> > Moreover, I think, that we should warn our users that some feature is
> very fresh and can have issues.
> >
> > So, Why Kafka and Spark is good enough to have unstable API and Ignite
> not? :)
> >
> >>  9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]>
> написал(а):
> >>
> >>  Conversation shifted into an unintended direction, but I agree.
> >>
> >>  I think that if API can (or will) be changed then it should be
> deprecated.
> >>  For that
> >>  we can introduce @IgniteDeprecated that will contain Ignite version
> when
> >>  API is planned to be removed. Otherwise it's either stable or
> experimental.
> >>  Having officially "unstable" features doesn't sound good for product
> >>  reputation.
> >>
> >>  As for the modularization - I'm all for this idea. If we don't force
> >>  ourselves to
> >>  organize code properly then we'll end up with the same problems as we
> have
> >>  in the current code base. And this way there's a hope of having good
> tests
> >>  that can be completed in minutes, not hours. At least new ones.
> >>
> >>  BTW, did we have any discussions about dependency injection and all
> this
> >>  stuff?
> >>  Seems like a related topic to me.
> >>
> >>  ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky
> <[hidden email]>:
> >>
> >>>  Hello Nikolay, if i find out introduced features structure in some
> >>>  project, i would prefer to choose different one )
> >>>
> >>>>>>  Hello, Alexey.
> >>>>>>
> >>>>>>  Think we can extend our @IgniteExperimental annotation.
> >>>>>>
> >>>>>>  `@IgniteExperimental` - mark features that are truly experimental
> and
> >>>  can be completely removed in future releases.
> >>>>>>  `@NotRecommended` - mark features that widely adopted by the users
> but
> >>>  implemented wrong or have known issues that can’t be fixed.
> >>>>>>  `@NotStable` - mark features supported by community but API not
> stable
> >>>  and can be reworked in the next release.
> >>>>>>  `@Stable` - mark features that are completely OK and here to stay.
> >>>>>>
> >>>>>>  We should output notes about these annotations in the JavaDoc,
> also.
> >>>>>>  What do you think?
> >>>>>>
> >>>>>>>  8 дек. 2020 г., в 12:49, Alexey Goncharuk <
> >>>  [hidden email] > написал(а):
> >>>>>>>  Igniters,
> >>>>>>>
> >>>>>>>  I want to tackle the topic of modules structure in Ignite 3. So
> far,
> >>>  the
> >>>>>>>  modules in Ignite are mostly defined intuitively which leads to
> some
> >>>>>>>  complications:
> >>>>>>>
> >>>>>>>  - Ignite public API is separated from the rest of the code only by
> >>>>>>>  package name. This leads to private classes leaking to public API
> >>>  which is
> >>>>>>>  very hard to catch even during the review process (we missed a
> bunch
> >>>  of
> >>>>>>>  such leaks for new metrics API [1] and I remember this happening
> for
> >>>  almost
> >>>>>>>  every SPI)
> >>>>>>>  - Classes from 'internal' packages are considered to be 'free for
> >>>  grabs'
> >>>>>>>  in every place of the code. This leads to tight coupling and
> >>>  abstraction
> >>>>>>>  leakage in the code. An example of such a case - an often cast of
> >>>>>>>  WALPointer to FileWALPointer, so that the community decided to get
> >>>  rid of
> >>>>>>>  the WALPointer interface altogether [2]
> >>>>>>>  - Overall code complexity. Because of the lack of inter-module
> >>>>>>>  interaction rules, we are free to add new methods and callbacks
> to any
> >>>>>>>  class, which leads to duplicating entities and verbose
> interfaces. A
> >>>  good
> >>>>>>>  example of this is the clear duplication of methods in
> >>>>>>>  IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore
> [3]
> >>>>>>>
> >>>>>>>  I think we need to work out some rules that will help us define
> and
> >>>  control
> >>>>>>>  both Ignite public API and module internal API which still
> defines a
> >>>  clear
> >>>>>>>  contract for other modules. Some ideas:
> >>>>>>>
> >>>>>>>  - Perhaps we can move all user public classed and interfaces to an
> >>>>>>>  Ignite-API module which will have no dependencies on
> implementation
> >>>>>>>  modules. This will prevent private classes from leaking to the API
> >>>  module.
> >>>>>>>  - We need somehow define which classes from a module are exposed
> to
> >>>>>>>  other modules, and which classes are left for module-private
> usage.
> >>>  Maybe
> >>>>>>>  Java's jigsaw will help us here, but maybe we will be ok with just
> >>>  more
> >>>>>>>  strict java access modifiers usage :) The idea here is that a
> module
> >>>  should
> >>>>>>>  never touch a dependent module's private classes, ever. The
> exported
> >>>>>>>  classes and interfaces are still free to be modified between
> >>>  releases, as
> >>>>>>>  long as it is not a user public API.
> >>>>>>>  - A module should be logically complete, thus it may be
> beneficial if
> >>>>>>>  module name matches with the code package it provides (e.g.
> >>>  configuration
> >>>>>>>  -> org.apache.ignite.configuration, replication ->
> >>>>>>>  org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
> >>>>>>>
> >>>>>>>  Any other principles/rules we can apply to make the code structure
> >>>  more
> >>>>>>>  concise? Thoughts?
> >>>>>>>
> >>>>>>>  --AG
> >>>>>>>
> >>>>>>>  [1] https://issues.apache.org/jira/browse/IGNITE-12552
> >>>>>>>  [2] https://issues.apache.org/jira/browse/IGNITE-13513
> >>>>>>>  [3] https://issues.apache.org/jira/browse/IGNITE-13220
> >>
> >>  --
> >>  Sincerely yours,
> >>  Ivan Bessonov
>

[Anton Kalashnikov]

Ilya, it is a good point about jar name. If it really unresolvable problem we should leave modules name as is(I do not insist on renaming it was just a proposal)

I also not really got you point about 'zillion JARs'. Do you think it will be a problem if the number of JARs increases a little? If so can you clarify it with some specific examples?

-- 
Best regards,
Anton Kalashnikov



09.12.2020, 18:12, "Ilya Kasnacheev" <[hidden email]>:

> Hello!
>
> When you do mvn dependencies:copy, you will now end up with
> "ignite-core-X.Y.Z.jar", "ignite-indexing-X.Y.Z.jar"
> But if we remove "ignite" from artifact name, user will end up with cryptic
> "configuration.jar", "raft.jar", etc, etc.
>
> Remember that group name is discarded in file names. Therefore, every
> project out there keeps stuff like "spring-" or "hibernate-" in artifact
> name.
>
> I also hope that our users will not end up with zillion JARs.
>
> Regards,
> --
> Ilya Kasnacheev
>
> ср, 9 дек. 2020 г. в 16:16, Anton Kalashnikov <[hidden email]>:
>
>>  Hello,
>>
>>  I totally agree that we should start to think about the module
>>  organization.
>>  I suggest making the new confluence page where we define new rules on how
>>  to develop modules.
>>  In my opinion, at least, the following topics should be covered there(it
>>  makes sense to discuss every topic separately, not here):
>>  * In which cases new modules are required
>>  * The naming of modules, packages
>>  * Class dependency management ( inversion of control, no more context)
>>  * Test organization ( module contains only unit, module tests. All
>>  integration tests are in the extra module)
>>  * Feature/Module lifecycle - experimental, stable, unstable, deprecated
>>
>>  Let's get back to the original topic. I agree with the package naming
>>  rule:
>>  if the module name is configuration the package name should be
>>  org.apache.ignite.configuration and only after that any other subpackages.
>>  Also, I don't sure that we need ignite- prefix in the module name because
>>  it doesn't have any extra information:
>>
>>    <groupId>org.apache.ignite</groupId>
>>    <artifactId>ignite-configuration</artifactId>
>>
>>  we don't lose anything if convert it to
>>
>>    <groupId>org.apache.ignite</groupId>
>>    <artifactId>configuration</artifactId>
>>
>>  I also hope that jigsaw can help us somehow with class visibility.
>>  But if not we can take agreement that for example 'internal' package -
>>  shouldn't be touched outside of the module -
>>  of course, using the proper class access level is also the solution(where
>>  it is possible)
>>  org.apache.ignite.configuration.Configurator // it has access to the
>>  internal package but it shouldn't return any class from the internal
>>  package only from the public one.
>>  org.apache.ignite.configuration.DynamicProperty //interface
>>  org.apache.ignite.configuration.internal.properties.IntDynamicProperty
>>
>>  User API makes sense only for one end module - ignite(or ignite-core)
>>  which depends on all other modules
>>  and doing some integration(adapters) and provide final API for the user.
>>  So I agree that separated module Ignite-API with zero dependencies will be
>>  a good solution.
>>
>>  configuration module:
>>    Configurator.baseline().enabled() -> DynamicProperties
>>
>>  ignite-api module:
>>    BaselineConfiguration.enabled() -> boolean //interface
>>
>>  ignite module:
>>    BaselineConfigurationImpl implements BaselineConfiguration{
>>      Configurator configurator;
>>      public boolean enabled(){
>>         return configurator.baseline().enabled().value();
>>      }
>>    }
>>
>>  So maybe my example is not so good. But I want to show that end-user API
>>  will be defined only in ignite-api
>>  and you need to adapt it in ignite module which leads to some
>>  overhead(especially in my example)
>>   but it makes development pretty manageable/predictable -
>>  you can easily implement a new module without any worries that user starts
>>  to use it.
>>  It will be available only after making changes in ignite-api.
>>  The major advantage here is the small size of ignite-api which allows to
>>  carefully review every change
>>  which allows keeping ignite API in better quality(I hope at least)
>>
>>  Nikolay, maybe is it better to discuss your question on a separate topic?
>>  Because looks like it is a pretty discussable topic.
>>
>>  --
>>  Best regards,
>>  Anton Kalashnikov
>>
>>  09.12.2020, 10:31, "Nikolay Izhikov" <[hidden email]>:
>>  > Hello, Zhenya, Ivan.
>>  >
>>  >> Hello Nikolay, if i find out introduced features structure in some
>>  project, i would prefer to choose different one )
>>  >
>>  > Many, of the real world users disagree with you.
>>  > Please, take a look at some examples from widely used projects:
>>  >
>>  > Kafka -
>>  https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
>>  > - Stable, Evolving, Unstable
>>  >
>>  > Spark -
>>  https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
>>  > - AlphaComponent, DeveloperApi, Evolving, Experimental, Private,
>>  Stable, Unstable
>>  >
>>  >> Having officially "unstable" features doesn't sound good for product
>>  reputation.
>>  >
>>  > Can’t agree with you.
>>  >
>>  > Forcing ourselves to make perfect API from the first try we just put too
>>  much pressure on every decision.
>>  > Every developer making mistakes.
>>  > The product is evolving and the API too - it’s totally OK.
>>  >
>>  > For every new feature time required to be adopted and used in real-world
>>  production.
>>  > I believe, slight API changes is totally fine for early adopters.
>>  > Moreover, I think, that we should warn our users that some feature is
>>  very fresh and can have issues.
>>  >
>>  > So, Why Kafka and Spark is good enough to have unstable API and Ignite
>>  not? :)
>>  >
>>  >> 9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]>
>>  написал(а):
>>  >>
>>  >> Conversation shifted into an unintended direction, but I agree.
>>  >>
>>  >> I think that if API can (or will) be changed then it should be
>>  deprecated.
>>  >> For that
>>  >> we can introduce @IgniteDeprecated that will contain Ignite version
>>  when
>>  >> API is planned to be removed. Otherwise it's either stable or
>>  experimental.
>>  >> Having officially "unstable" features doesn't sound good for product
>>  >> reputation.
>>  >>
>>  >> As for the modularization - I'm all for this idea. If we don't force
>>  >> ourselves to
>>  >> organize code properly then we'll end up with the same problems as we
>>  have
>>  >> in the current code base. And this way there's a hope of having good
>>  tests
>>  >> that can be completed in minutes, not hours. At least new ones.
>>  >>
>>  >> BTW, did we have any discussions about dependency injection and all
>>  this
>>  >> stuff?
>>  >> Seems like a related topic to me.
>>  >>
>>  >> ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky
>>  <[hidden email]>:
>>  >>
>>  >>> Hello Nikolay, if i find out introduced features structure in some
>>  >>> project, i would prefer to choose different one )
>>  >>>
>>  >>>>>> Hello, Alexey.
>>  >>>>>>
>>  >>>>>> Think we can extend our @IgniteExperimental annotation.
>>  >>>>>>
>>  >>>>>> `@IgniteExperimental` - mark features that are truly experimental
>>  and
>>  >>> can be completely removed in future releases.
>>  >>>>>> `@NotRecommended` - mark features that widely adopted by the users
>>  but
>>  >>> implemented wrong or have known issues that can’t be fixed.
>>  >>>>>> `@NotStable` - mark features supported by community but API not
>>  stable
>>  >>> and can be reworked in the next release.
>>  >>>>>> `@Stable` - mark features that are completely OK and here to stay.
>>  >>>>>>
>>  >>>>>> We should output notes about these annotations in the JavaDoc,
>>  also.
>>  >>>>>> What do you think?
>>  >>>>>>
>>  >>>>>>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk <
>>  >>> [hidden email] > написал(а):
>>  >>>>>>> Igniters,
>>  >>>>>>>
>>  >>>>>>> I want to tackle the topic of modules structure in Ignite 3. So
>>  far,
>>  >>> the
>>  >>>>>>> modules in Ignite are mostly defined intuitively which leads to
>>  some
>>  >>>>>>> complications:
>>  >>>>>>>
>>  >>>>>>> - Ignite public API is separated from the rest of the code only by
>>  >>>>>>> package name. This leads to private classes leaking to public API
>>  >>> which is
>>  >>>>>>> very hard to catch even during the review process (we missed a
>>  bunch
>>  >>> of
>>  >>>>>>> such leaks for new metrics API [1] and I remember this happening
>>  for
>>  >>> almost
>>  >>>>>>> every SPI)
>>  >>>>>>> - Classes from 'internal' packages are considered to be 'free for
>>  >>> grabs'
>>  >>>>>>> in every place of the code. This leads to tight coupling and
>>  >>> abstraction
>>  >>>>>>> leakage in the code. An example of such a case - an often cast of
>>  >>>>>>> WALPointer to FileWALPointer, so that the community decided to get
>>  >>> rid of
>>  >>>>>>> the WALPointer interface altogether [2]
>>  >>>>>>> - Overall code complexity. Because of the lack of inter-module
>>  >>>>>>> interaction rules, we are free to add new methods and callbacks
>>  to any
>>  >>>>>>> class, which leads to duplicating entities and verbose
>>  interfaces. A
>>  >>> good
>>  >>>>>>> example of this is the clear duplication of methods in
>>  >>>>>>> IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore
>>  [3]
>>  >>>>>>>
>>  >>>>>>> I think we need to work out some rules that will help us define
>>  and
>>  >>> control
>>  >>>>>>> both Ignite public API and module internal API which still
>>  defines a
>>  >>> clear
>>  >>>>>>> contract for other modules. Some ideas:
>>  >>>>>>>
>>  >>>>>>> - Perhaps we can move all user public classed and interfaces to an
>>  >>>>>>> Ignite-API module which will have no dependencies on
>>  implementation
>>  >>>>>>> modules. This will prevent private classes from leaking to the API
>>  >>> module.
>>  >>>>>>> - We need somehow define which classes from a module are exposed
>>  to
>>  >>>>>>> other modules, and which classes are left for module-private
>>  usage.
>>  >>> Maybe
>>  >>>>>>> Java's jigsaw will help us here, but maybe we will be ok with just
>>  >>> more
>>  >>>>>>> strict java access modifiers usage :) The idea here is that a
>>  module
>>  >>> should
>>  >>>>>>> never touch a dependent module's private classes, ever. The
>>  exported
>>  >>>>>>> classes and interfaces are still free to be modified between
>>  >>> releases, as
>>  >>>>>>> long as it is not a user public API.
>>  >>>>>>> - A module should be logically complete, thus it may be
>>  beneficial if
>>  >>>>>>> module name matches with the code package it provides (e.g.
>>  >>> configuration
>>  >>>>>>> -> org.apache.ignite.configuration, replication ->
>>  >>>>>>> org.apache.ignite.replication, raft->org.apache.ignite.raft, etc)
>>  >>>>>>>
>>  >>>>>>> Any other principles/rules we can apply to make the code structure
>>  >>> more
>>  >>>>>>> concise? Thoughts?
>>  >>>>>>>
>>  >>>>>>> --AG
>>  >>>>>>>
>>  >>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12552
>>  >>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-13513
>>  >>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13220
>>  >>
>>  >> --
>>  >> Sincerely yours,
>>  >> Ivan Bessonov

[Ilya Kasnacheev]

Hello!

I think it is a strong side of Apache Ignite that it has a low number of
JARs and minimal dependencies. It may be less relevant now than 10 years
ago, but it's still a plus.

Not sure how fast it would evaporate as the number of JARs and dependencies
grows.

Regards,
--
Ilya Kasnacheev


чт, 10 дек. 2020 г. в 12:28, Anton Kalashnikov <[hidden email]>:

> Ilya, it is a good point about jar name. If it really unresolvable problem
> we should leave modules name as is(I do not insist on renaming it was just
> a proposal)
>
> I also not really got you point about 'zillion JARs'. Do you think it will
> be a problem if the number of JARs increases a little? If so can you
> clarify it with some specific examples?
>
> --
> Best regards,
> Anton Kalashnikov
>
>
>
> 09.12.2020, 18:12, "Ilya Kasnacheev" <[hidden email]>:
> > Hello!
> >
> > When you do mvn dependencies:copy, you will now end up with
> > "ignite-core-X.Y.Z.jar", "ignite-indexing-X.Y.Z.jar"
> > But if we remove "ignite" from artifact name, user will end up with
> cryptic
> > "configuration.jar", "raft.jar", etc, etc.
> >
> > Remember that group name is discarded in file names. Therefore, every
> > project out there keeps stuff like "spring-" or "hibernate-" in artifact
> > name.
> >
> > I also hope that our users will not end up with zillion JARs.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> > ср, 9 дек. 2020 г. в 16:16, Anton Kalashnikov <[hidden email]>:
> >
> >>  Hello,
> >>
> >>  I totally agree that we should start to think about the module
> >>  organization.
> >>  I suggest making the new confluence page where we define new rules on
> how
> >>  to develop modules.
> >>  In my opinion, at least, the following topics should be covered
> there(it
> >>  makes sense to discuss every topic separately, not here):
> >>  * In which cases new modules are required
> >>  * The naming of modules, packages
> >>  * Class dependency management ( inversion of control, no more context)
> >>  * Test organization ( module contains only unit, module tests. All
> >>  integration tests are in the extra module)
> >>  * Feature/Module lifecycle - experimental, stable, unstable, deprecated
> >>
> >>  Let's get back to the original topic. I agree with the package naming
> >>  rule:
> >>  if the module name is configuration the package name should be
> >>  org.apache.ignite.configuration and only after that any other
> subpackages.
> >>  Also, I don't sure that we need ignite- prefix in the module name
> because
> >>  it doesn't have any extra information:
> >>
> >>    <groupId>org.apache.ignite</groupId>
> >>    <artifactId>ignite-configuration</artifactId>
> >>
> >>  we don't lose anything if convert it to
> >>
> >>    <groupId>org.apache.ignite</groupId>
> >>    <artifactId>configuration</artifactId>
> >>
> >>  I also hope that jigsaw can help us somehow with class visibility.
> >>  But if not we can take agreement that for example 'internal' package -
> >>  shouldn't be touched outside of the module -
> >>  of course, using the proper class access level is also the
> solution(where
> >>  it is possible)
> >>  org.apache.ignite.configuration.Configurator // it has access to the
> >>  internal package but it shouldn't return any class from the internal
> >>  package only from the public one.
> >>  org.apache.ignite.configuration.DynamicProperty //interface
> >>  org.apache.ignite.configuration.internal.properties.IntDynamicProperty
> >>
> >>  User API makes sense only for one end module - ignite(or ignite-core)
> >>  which depends on all other modules
> >>  and doing some integration(adapters) and provide final API for the
> user.
> >>  So I agree that separated module Ignite-API with zero dependencies
> will be
> >>  a good solution.
> >>
> >>  configuration module:
> >>    Configurator.baseline().enabled() -> DynamicProperties
> >>
> >>  ignite-api module:
> >>    BaselineConfiguration.enabled() -> boolean //interface
> >>
> >>  ignite module:
> >>    BaselineConfigurationImpl implements BaselineConfiguration{
> >>      Configurator configurator;
> >>      public boolean enabled(){
> >>         return configurator.baseline().enabled().value();
> >>      }
> >>    }
> >>
> >>  So maybe my example is not so good. But I want to show that end-user
> API
> >>  will be defined only in ignite-api
> >>  and you need to adapt it in ignite module which leads to some
> >>  overhead(especially in my example)
> >>   but it makes development pretty manageable/predictable -
> >>  you can easily implement a new module without any worries that user
> starts
> >>  to use it.
> >>  It will be available only after making changes in ignite-api.
> >>  The major advantage here is the small size of ignite-api which allows
> to
> >>  carefully review every change
> >>  which allows keeping ignite API in better quality(I hope at least)
> >>
> >>  Nikolay, maybe is it better to discuss your question on a separate
> topic?
> >>  Because looks like it is a pretty discussable topic.
> >>
> >>  --
> >>  Best regards,
> >>  Anton Kalashnikov
> >>
> >>  09.12.2020, 10:31, "Nikolay Izhikov" <[hidden email]>:
> >>  > Hello, Zhenya, Ivan.
> >>  >
> >>  >> Hello Nikolay, if i find out introduced features structure in some
> >>  project, i would prefer to choose different one )
> >>  >
> >>  > Many, of the real world users disagree with you.
> >>  > Please, take a look at some examples from widely used projects:
> >>  >
> >>  > Kafka -
> >>
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
> >>  > - Stable, Evolving, Unstable
> >>  >
> >>  > Spark -
> >>
> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
> >>  > - AlphaComponent, DeveloperApi, Evolving, Experimental, Private,
> >>  Stable, Unstable
> >>  >
> >>  >> Having officially "unstable" features doesn't sound good for product
> >>  reputation.
> >>  >
> >>  > Can’t agree with you.
> >>  >
> >>  > Forcing ourselves to make perfect API from the first try we just put
> too
> >>  much pressure on every decision.
> >>  > Every developer making mistakes.
> >>  > The product is evolving and the API too - it’s totally OK.
> >>  >
> >>  > For every new feature time required to be adopted and used in
> real-world
> >>  production.
> >>  > I believe, slight API changes is totally fine for early adopters.
> >>  > Moreover, I think, that we should warn our users that some feature is
> >>  very fresh and can have issues.
> >>  >
> >>  > So, Why Kafka and Spark is good enough to have unstable API and
> Ignite
> >>  not? :)
> >>  >
> >>  >> 9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]>
> >>  написал(а):
> >>  >>
> >>  >> Conversation shifted into an unintended direction, but I agree.
> >>  >>
> >>  >> I think that if API can (or will) be changed then it should be
> >>  deprecated.
> >>  >> For that
> >>  >> we can introduce @IgniteDeprecated that will contain Ignite version
> >>  when
> >>  >> API is planned to be removed. Otherwise it's either stable or
> >>  experimental.
> >>  >> Having officially "unstable" features doesn't sound good for product
> >>  >> reputation.
> >>  >>
> >>  >> As for the modularization - I'm all for this idea. If we don't force
> >>  >> ourselves to
> >>  >> organize code properly then we'll end up with the same problems as
> we
> >>  have
> >>  >> in the current code base. And this way there's a hope of having good
> >>  tests
> >>  >> that can be completed in minutes, not hours. At least new ones.
> >>  >>
> >>  >> BTW, did we have any discussions about dependency injection and all
> >>  this
> >>  >> stuff?
> >>  >> Seems like a related topic to me.
> >>  >>
> >>  >> ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky
> >>  <[hidden email]>:
> >>  >>
> >>  >>> Hello Nikolay, if i find out introduced features structure in some
> >>  >>> project, i would prefer to choose different one )
> >>  >>>
> >>  >>>>>> Hello, Alexey.
> >>  >>>>>>
> >>  >>>>>> Think we can extend our @IgniteExperimental annotation.
> >>  >>>>>>
> >>  >>>>>> `@IgniteExperimental` - mark features that are truly
> experimental
> >>  and
> >>  >>> can be completely removed in future releases.
> >>  >>>>>> `@NotRecommended` - mark features that widely adopted by the
> users
> >>  but
> >>  >>> implemented wrong or have known issues that can’t be fixed.
> >>  >>>>>> `@NotStable` - mark features supported by community but API not
> >>  stable
> >>  >>> and can be reworked in the next release.
> >>  >>>>>> `@Stable` - mark features that are completely OK and here to
> stay.
> >>  >>>>>>
> >>  >>>>>> We should output notes about these annotations in the JavaDoc,
> >>  also.
> >>  >>>>>> What do you think?
> >>  >>>>>>
> >>  >>>>>>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk <
> >>  >>> [hidden email] > написал(а):
> >>  >>>>>>> Igniters,
> >>  >>>>>>>
> >>  >>>>>>> I want to tackle the topic of modules structure in Ignite 3. So
> >>  far,
> >>  >>> the
> >>  >>>>>>> modules in Ignite are mostly defined intuitively which leads to
> >>  some
> >>  >>>>>>> complications:
> >>  >>>>>>>
> >>  >>>>>>> - Ignite public API is separated from the rest of the code
> only by
> >>  >>>>>>> package name. This leads to private classes leaking to public
> API
> >>  >>> which is
> >>  >>>>>>> very hard to catch even during the review process (we missed a
> >>  bunch
> >>  >>> of
> >>  >>>>>>> such leaks for new metrics API [1] and I remember this
> happening
> >>  for
> >>  >>> almost
> >>  >>>>>>> every SPI)
> >>  >>>>>>> - Classes from 'internal' packages are considered to be 'free
> for
> >>  >>> grabs'
> >>  >>>>>>> in every place of the code. This leads to tight coupling and
> >>  >>> abstraction
> >>  >>>>>>> leakage in the code. An example of such a case - an often cast
> of
> >>  >>>>>>> WALPointer to FileWALPointer, so that the community decided to
> get
> >>  >>> rid of
> >>  >>>>>>> the WALPointer interface altogether [2]
> >>  >>>>>>> - Overall code complexity. Because of the lack of inter-module
> >>  >>>>>>> interaction rules, we are free to add new methods and callbacks
> >>  to any
> >>  >>>>>>> class, which leads to duplicating entities and verbose
> >>  interfaces. A
> >>  >>> good
> >>  >>>>>>> example of this is the clear duplication of methods in
> >>  >>>>>>> IgniteCacheOffheapManager and
> IgniteCacheOffheapManager.DataStore
> >>  [3]
> >>  >>>>>>>
> >>  >>>>>>> I think we need to work out some rules that will help us define
> >>  and
> >>  >>> control
> >>  >>>>>>> both Ignite public API and module internal API which still
> >>  defines a
> >>  >>> clear
> >>  >>>>>>> contract for other modules. Some ideas:
> >>  >>>>>>>
> >>  >>>>>>> - Perhaps we can move all user public classed and interfaces
> to an
> >>  >>>>>>> Ignite-API module which will have no dependencies on
> >>  implementation
> >>  >>>>>>> modules. This will prevent private classes from leaking to the
> API
> >>  >>> module.
> >>  >>>>>>> - We need somehow define which classes from a module are
> exposed
> >>  to
> >>  >>>>>>> other modules, and which classes are left for module-private
> >>  usage.
> >>  >>> Maybe
> >>  >>>>>>> Java's jigsaw will help us here, but maybe we will be ok with
> just
> >>  >>> more
> >>  >>>>>>> strict java access modifiers usage :) The idea here is that a
> >>  module
> >>  >>> should
> >>  >>>>>>> never touch a dependent module's private classes, ever. The
> >>  exported
> >>  >>>>>>> classes and interfaces are still free to be modified between
> >>  >>> releases, as
> >>  >>>>>>> long as it is not a user public API.
> >>  >>>>>>> - A module should be logically complete, thus it may be
> >>  beneficial if
> >>  >>>>>>> module name matches with the code package it provides (e.g.
> >>  >>> configuration
> >>  >>>>>>> -> org.apache.ignite.configuration, replication ->
> >>  >>>>>>> org.apache.ignite.replication, raft->org.apache.ignite.raft,
> etc)
> >>  >>>>>>>
> >>  >>>>>>> Any other principles/rules we can apply to make the code
> structure
> >>  >>> more
> >>  >>>>>>> concise? Thoughts?
> >>  >>>>>>>
> >>  >>>>>>> --AG
> >>  >>>>>>>
> >>  >>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12552
> >>  >>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-13513
> >>  >>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13220
> >>  >>
> >>  >> --
> >>  >> Sincerely yours,
> >>  >> Ivan Bessonov
>

[Valentin Kulichenko]

Ilya,

I think common sense is our only friend here :) The low number of
dependencies is indeed one of the advantages of Ignite. On the other hand,
the "zero dependency" approach we've been trying to follow for the last
several years sometimes forces us to implement, stabilize and maintain a
whole lot of code that we potentially could reuse from somewhere else.

We should always try to minimize the number of dependencies, especially for
the core part of the product. But we should also reuse existing code where
it is reasonable.

-Val


On Fri, Dec 11, 2020 at 3:26 AM Ilya Kasnacheev <[hidden email]>
wrote:

> Hello!
>
> I think it is a strong side of Apache Ignite that it has a low number of
> JARs and minimal dependencies. It may be less relevant now than 10 years
> ago, but it's still a plus.
>
> Not sure how fast it would evaporate as the number of JARs and dependencies
> grows.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 10 дек. 2020 г. в 12:28, Anton Kalashnikov <[hidden email]>:
>
> > Ilya, it is a good point about jar name. If it really unresolvable
> problem
> > we should leave modules name as is(I do not insist on renaming it was
> just
> > a proposal)
> >
> > I also not really got you point about 'zillion JARs'. Do you think it
> will
> > be a problem if the number of JARs increases a little? If so can you
> > clarify it with some specific examples?
> >
> > --
> > Best regards,
> > Anton Kalashnikov
> >
> >
> >
> > 09.12.2020, 18:12, "Ilya Kasnacheev" <[hidden email]>:
> > > Hello!
> > >
> > > When you do mvn dependencies:copy, you will now end up with
> > > "ignite-core-X.Y.Z.jar", "ignite-indexing-X.Y.Z.jar"
> > > But if we remove "ignite" from artifact name, user will end up with
> > cryptic
> > > "configuration.jar", "raft.jar", etc, etc.
> > >
> > > Remember that group name is discarded in file names. Therefore, every
> > > project out there keeps stuff like "spring-" or "hibernate-" in
> artifact
> > > name.
> > >
> > > I also hope that our users will not end up with zillion JARs.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > > ср, 9 дек. 2020 г. в 16:16, Anton Kalashnikov <[hidden email]>:
> > >
> > >>  Hello,
> > >>
> > >>  I totally agree that we should start to think about the module
> > >>  organization.
> > >>  I suggest making the new confluence page where we define new rules on
> > how
> > >>  to develop modules.
> > >>  In my opinion, at least, the following topics should be covered
> > there(it
> > >>  makes sense to discuss every topic separately, not here):
> > >>  * In which cases new modules are required
> > >>  * The naming of modules, packages
> > >>  * Class dependency management ( inversion of control, no more
> context)
> > >>  * Test organization ( module contains only unit, module tests. All
> > >>  integration tests are in the extra module)
> > >>  * Feature/Module lifecycle - experimental, stable, unstable,
> deprecated
> > >>
> > >>  Let's get back to the original topic. I agree with the package naming
> > >>  rule:
> > >>  if the module name is configuration the package name should be
> > >>  org.apache.ignite.configuration and only after that any other
> > subpackages.
> > >>  Also, I don't sure that we need ignite- prefix in the module name
> > because
> > >>  it doesn't have any extra information:
> > >>
> > >>    <groupId>org.apache.ignite</groupId>
> > >>    <artifactId>ignite-configuration</artifactId>
> > >>
> > >>  we don't lose anything if convert it to
> > >>
> > >>    <groupId>org.apache.ignite</groupId>
> > >>    <artifactId>configuration</artifactId>
> > >>
> > >>  I also hope that jigsaw can help us somehow with class visibility.
> > >>  But if not we can take agreement that for example 'internal' package
> -
> > >>  shouldn't be touched outside of the module -
> > >>  of course, using the proper class access level is also the
> > solution(where
> > >>  it is possible)
> > >>  org.apache.ignite.configuration.Configurator // it has access to the
> > >>  internal package but it shouldn't return any class from the internal
> > >>  package only from the public one.
> > >>  org.apache.ignite.configuration.DynamicProperty //interface
> > >>
> org.apache.ignite.configuration.internal.properties.IntDynamicProperty
> > >>
> > >>  User API makes sense only for one end module - ignite(or ignite-core)
> > >>  which depends on all other modules
> > >>  and doing some integration(adapters) and provide final API for the
> > user.
> > >>  So I agree that separated module Ignite-API with zero dependencies
> > will be
> > >>  a good solution.
> > >>
> > >>  configuration module:
> > >>    Configurator.baseline().enabled() -> DynamicProperties
> > >>
> > >>  ignite-api module:
> > >>    BaselineConfiguration.enabled() -> boolean //interface
> > >>
> > >>  ignite module:
> > >>    BaselineConfigurationImpl implements BaselineConfiguration{
> > >>      Configurator configurator;
> > >>      public boolean enabled(){
> > >>         return configurator.baseline().enabled().value();
> > >>      }
> > >>    }
> > >>
> > >>  So maybe my example is not so good. But I want to show that end-user
> > API
> > >>  will be defined only in ignite-api
> > >>  and you need to adapt it in ignite module which leads to some
> > >>  overhead(especially in my example)
> > >>   but it makes development pretty manageable/predictable -
> > >>  you can easily implement a new module without any worries that user
> > starts
> > >>  to use it.
> > >>  It will be available only after making changes in ignite-api.
> > >>  The major advantage here is the small size of ignite-api which allows
> > to
> > >>  carefully review every change
> > >>  which allows keeping ignite API in better quality(I hope at least)
> > >>
> > >>  Nikolay, maybe is it better to discuss your question on a separate
> > topic?
> > >>  Because looks like it is a pretty discussable topic.
> > >>
> > >>  --
> > >>  Best regards,
> > >>  Anton Kalashnikov
> > >>
> > >>  09.12.2020, 10:31, "Nikolay Izhikov" <[hidden email]>:
> > >>  > Hello, Zhenya, Ivan.
> > >>  >
> > >>  >> Hello Nikolay, if i find out introduced features structure in some
> > >>  project, i would prefer to choose different one )
> > >>  >
> > >>  > Many, of the real world users disagree with you.
> > >>  > Please, take a look at some examples from widely used projects:
> > >>  >
> > >>  > Kafka -
> > >>
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
> > >>  > - Stable, Evolving, Unstable
> > >>  >
> > >>  > Spark -
> > >>
> >
> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
> > >>  > - AlphaComponent, DeveloperApi, Evolving, Experimental, Private,
> > >>  Stable, Unstable
> > >>  >
> > >>  >> Having officially "unstable" features doesn't sound good for
> product
> > >>  reputation.
> > >>  >
> > >>  > Can’t agree with you.
> > >>  >
> > >>  > Forcing ourselves to make perfect API from the first try we just
> put
> > too
> > >>  much pressure on every decision.
> > >>  > Every developer making mistakes.
> > >>  > The product is evolving and the API too - it’s totally OK.
> > >>  >
> > >>  > For every new feature time required to be adopted and used in
> > real-world
> > >>  production.
> > >>  > I believe, slight API changes is totally fine for early adopters.
> > >>  > Moreover, I think, that we should warn our users that some feature
> is
> > >>  very fresh and can have issues.
> > >>  >
> > >>  > So, Why Kafka and Spark is good enough to have unstable API and
> > Ignite
> > >>  not? :)
> > >>  >
> > >>  >> 9 дек. 2020 г., в 10:08, Ivan Bessonov <[hidden email]>
> > >>  написал(а):
> > >>  >>
> > >>  >> Conversation shifted into an unintended direction, but I agree.
> > >>  >>
> > >>  >> I think that if API can (or will) be changed then it should be
> > >>  deprecated.
> > >>  >> For that
> > >>  >> we can introduce @IgniteDeprecated that will contain Ignite
> version
> > >>  when
> > >>  >> API is planned to be removed. Otherwise it's either stable or
> > >>  experimental.
> > >>  >> Having officially "unstable" features doesn't sound good for
> product
> > >>  >> reputation.
> > >>  >>
> > >>  >> As for the modularization - I'm all for this idea. If we don't
> force
> > >>  >> ourselves to
> > >>  >> organize code properly then we'll end up with the same problems as
> > we
> > >>  have
> > >>  >> in the current code base. And this way there's a hope of having
> good
> > >>  tests
> > >>  >> that can be completed in minutes, not hours. At least new ones.
> > >>  >>
> > >>  >> BTW, did we have any discussions about dependency injection and
> all
> > >>  this
> > >>  >> stuff?
> > >>  >> Seems like a related topic to me.
> > >>  >>
> > >>  >> ср, 9 дек. 2020 г. в 09:47, Zhenya Stanilovsky
> > >>  <[hidden email]>:
> > >>  >>
> > >>  >>> Hello Nikolay, if i find out introduced features structure in
> some
> > >>  >>> project, i would prefer to choose different one )
> > >>  >>>
> > >>  >>>>>> Hello, Alexey.
> > >>  >>>>>>
> > >>  >>>>>> Think we can extend our @IgniteExperimental annotation.
> > >>  >>>>>>
> > >>  >>>>>> `@IgniteExperimental` - mark features that are truly
> > experimental
> > >>  and
> > >>  >>> can be completely removed in future releases.
> > >>  >>>>>> `@NotRecommended` - mark features that widely adopted by the
> > users
> > >>  but
> > >>  >>> implemented wrong or have known issues that can’t be fixed.
> > >>  >>>>>> `@NotStable` - mark features supported by community but API
> not
> > >>  stable
> > >>  >>> and can be reworked in the next release.
> > >>  >>>>>> `@Stable` - mark features that are completely OK and here to
> > stay.
> > >>  >>>>>>
> > >>  >>>>>> We should output notes about these annotations in the JavaDoc,
> > >>  also.
> > >>  >>>>>> What do you think?
> > >>  >>>>>>
> > >>  >>>>>>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk <
> > >>  >>> [hidden email] > написал(а):
> > >>  >>>>>>> Igniters,
> > >>  >>>>>>>
> > >>  >>>>>>> I want to tackle the topic of modules structure in Ignite 3.
> So
> > >>  far,
> > >>  >>> the
> > >>  >>>>>>> modules in Ignite are mostly defined intuitively which leads
> to
> > >>  some
> > >>  >>>>>>> complications:
> > >>  >>>>>>>
> > >>  >>>>>>> - Ignite public API is separated from the rest of the code
> > only by
> > >>  >>>>>>> package name. This leads to private classes leaking to public
> > API
> > >>  >>> which is
> > >>  >>>>>>> very hard to catch even during the review process (we missed
> a
> > >>  bunch
> > >>  >>> of
> > >>  >>>>>>> such leaks for new metrics API [1] and I remember this
> > happening
> > >>  for
> > >>  >>> almost
> > >>  >>>>>>> every SPI)
> > >>  >>>>>>> - Classes from 'internal' packages are considered to be 'free
> > for
> > >>  >>> grabs'
> > >>  >>>>>>> in every place of the code. This leads to tight coupling and
> > >>  >>> abstraction
> > >>  >>>>>>> leakage in the code. An example of such a case - an often
> cast
> > of
> > >>  >>>>>>> WALPointer to FileWALPointer, so that the community decided
> to
> > get
> > >>  >>> rid of
> > >>  >>>>>>> the WALPointer interface altogether [2]
> > >>  >>>>>>> - Overall code complexity. Because of the lack of
> inter-module
> > >>  >>>>>>> interaction rules, we are free to add new methods and
> callbacks
> > >>  to any
> > >>  >>>>>>> class, which leads to duplicating entities and verbose
> > >>  interfaces. A
> > >>  >>> good
> > >>  >>>>>>> example of this is the clear duplication of methods in
> > >>  >>>>>>> IgniteCacheOffheapManager and
> > IgniteCacheOffheapManager.DataStore
> > >>  [3]
> > >>  >>>>>>>
> > >>  >>>>>>> I think we need to work out some rules that will help us
> define
> > >>  and
> > >>  >>> control
> > >>  >>>>>>> both Ignite public API and module internal API which still
> > >>  defines a
> > >>  >>> clear
> > >>  >>>>>>> contract for other modules. Some ideas:
> > >>  >>>>>>>
> > >>  >>>>>>> - Perhaps we can move all user public classed and interfaces
> > to an
> > >>  >>>>>>> Ignite-API module which will have no dependencies on
> > >>  implementation
> > >>  >>>>>>> modules. This will prevent private classes from leaking to
> the
> > API
> > >>  >>> module.
> > >>  >>>>>>> - We need somehow define which classes from a module are
> > exposed
> > >>  to
> > >>  >>>>>>> other modules, and which classes are left for module-private
> > >>  usage.
> > >>  >>> Maybe
> > >>  >>>>>>> Java's jigsaw will help us here, but maybe we will be ok with
> > just
> > >>  >>> more
> > >>  >>>>>>> strict java access modifiers usage :) The idea here is that a
> > >>  module
> > >>  >>> should
> > >>  >>>>>>> never touch a dependent module's private classes, ever. The
> > >>  exported
> > >>  >>>>>>> classes and interfaces are still free to be modified between
> > >>  >>> releases, as
> > >>  >>>>>>> long as it is not a user public API.
> > >>  >>>>>>> - A module should be logically complete, thus it may be
> > >>  beneficial if
> > >>  >>>>>>> module name matches with the code package it provides (e.g.
> > >>  >>> configuration
> > >>  >>>>>>> -> org.apache.ignite.configuration, replication ->
> > >>  >>>>>>> org.apache.ignite.replication, raft->org.apache.ignite.raft,
> > etc)
> > >>  >>>>>>>
> > >>  >>>>>>> Any other principles/rules we can apply to make the code
> > structure
> > >>  >>> more
> > >>  >>>>>>> concise? Thoughts?
> > >>  >>>>>>>
> > >>  >>>>>>> --AG
> > >>  >>>>>>>
> > >>  >>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12552
> > >>  >>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-13513
> > >>  >>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13220
> > >>  >>
> > >>  >> --
> > >>  >> Sincerely yours,
> > >>  >> Ivan Bessonov
> >
>