Maven. Issues with flatten plugin

[Nikolay Izhikov-2]

Hello, Igniters.

We have `flatten-maven-plugin` in `parent/pom.xml` [1]
As far as I can understand it minimize pom.xml before it installed in repository.

It introduce some strange behavior in modules:

I must to enlist all test dependencies in my module to get tests work.
Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
It happens in `spark` [2] and `spark_2.10` [3] modules.
Now, when I want to enable testing of Spark Examples I has to enlist same dependencies in `examples/pom.xml`

It looks like a mess for me.

Please, help me:

1. Am I miss something and can make pom.xml much clearer?

2. Why we need to minimize pom.xml? It looks like other Apache project doesn't do it [5].

[1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
[2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
[3] https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150 
[4] https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
[5] http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom

[vveider]

Nikolay,


Can you describe your problem and desired behaviour more thoroughly, please?

> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]> wrote:
>
> Hello, Igniters.
>
> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> As far as I can understand it minimize pom.xml before it installed in repository.
>
> It introduce some strange behavior in modules:
>
> I must to enlist all test dependencies in my module to get tests work.
> Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
> It happens in `spark` [2] and `spark_2.10` [3] modules.
> Now, when I want to enable testing of Spark Examples I has to enlist same dependencies in `examples/pom.xml`
>
> It looks like a mess for me.
>
> Please, help me:
>
> 1. Am I miss something and can make pom.xml much clearer?
>
> 2. Why we need to minimize pom.xml? It looks like other Apache project doesn't do it [5].
>
> [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
> [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> [3] https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150 
> [4] https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom

[Nikolay Izhikov-2]

Hello, Petr.

> Can you describe your problem

In Ignite, maven doesn't include transitive dependencies to test classpath.
So we have enlist all dependencies in project pom.xml.

> desired behaviour

All I want is default maven behavior.

Please, take a look at pom.xml [1] from line 144.
There is a long list of dependencies with `test` scope.
Actually, all of them are available as a transitive dependency from `spark-core`.

We doesn't have to enlist them in every other project that doesn't use `flatten-plugin`.

[1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144

В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:

> Nikolay,
>
>
> Can you describe your problem and desired behaviour more thoroughly, please?
>
>
>
> > On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Hello, Igniters.
> >
> > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> > As far as I can understand it minimize pom.xml before it installed in repository.
> >
> > It introduce some strange behavior in modules:
> >
> > I must to enlist all test dependencies in my module to get tests work.
> > Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
> > It happens in `spark` [2] and `spark_2.10` [3] modules.
> > Now, when I want to enable testing of Spark Examples I has to enlist same dependencies in `examples/pom.xml`
> >
> > It looks like a mess for me.
> >
> > Please, help me:
> >
> > 1. Am I miss something and can make pom.xml much clearer?
> >
> > 2. Why we need to minimize pom.xml? It looks like other Apache project doesn't do it [5].
> >
> > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
> > [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> > [3] https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150 
> > [4] https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
> > [5] http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
>
>

[Andrey Novikov-3]

Nikolay,

As I remember we use `flatten-maven-plugin` for flattening and
removing parent relationship in deployed artifacts (parent pom does
not deploy to repository and in first versions of build, version was
stored in variable in parent pom)

On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]> wrote:

> Hello, Petr.
>
>> Can you describe your problem
>
> In Ignite, maven doesn't include transitive dependencies to test classpath.
> So we have enlist all dependencies in project pom.xml.
>
>> desired behaviour
>
> All I want is default maven behavior.
>
> Please, take a look at pom.xml [1] from line 144.
> There is a long list of dependencies with `test` scope.
> Actually, all of them are available as a transitive dependency from `spark-core`.
>
> We doesn't have to enlist them in every other project that doesn't use `flatten-plugin`.
>
> [1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>
> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> Nikolay,
>>
>>
>> Can you describe your problem and desired behaviour more thoroughly, please?
>>
>>
>>
>> > On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]> wrote:
>> >
>> > Hello, Igniters.
>> >
>> > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> > As far as I can understand it minimize pom.xml before it installed in repository.
>> >
>> > It introduce some strange behavior in modules:
>> >
>> > I must to enlist all test dependencies in my module to get tests work.
>> > Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
>> > It happens in `spark` [2] and `spark_2.10` [3] modules.
>> > Now, when I want to enable testing of Spark Examples I has to enlist same dependencies in `examples/pom.xml`
>> >
>> > It looks like a mess for me.
>> >
>> > Please, help me:
>> >
>> > 1. Am I miss something and can make pom.xml much clearer?
>> >
>> > 2. Why we need to minimize pom.xml? It looks like other Apache project doesn't do it [5].
>> >
>> > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
>> > [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>> > [3] https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
>> > [4] https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
>> > [5] http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
>>
>>

[Nikolay Izhikov-2]

Hello, Andrey.

Thanks for an answer.

> As I remember we use `flatten-maven-plugin` for flattening and removing parent relationship in deployed artifacts

So we need it only in `release` profile?

I found some earlier discussion about plugin [1]

> in first versions of build, version was stored in variable in parent pom

Got it. Do we need this plugin now?

[1] http://apache-ignite-developers.2346864.n4.nabble.com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html

В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:

> Nikolay,
>
> As I remember we use `flatten-maven-plugin` for flattening and
> removing parent relationship in deployed artifacts (parent pom does
> not deploy to repository and in first versions of build, version was
> stored in variable in parent pom)
>
> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]> wrote:
> > Hello, Petr.
> >
> > > Can you describe your problem
> >
> > In Ignite, maven doesn't include transitive dependencies to test classpath.
> > So we have enlist all dependencies in project pom.xml.
> >
> > > desired behaviour
> >
> > All I want is default maven behavior.
> >
> > Please, take a look at pom.xml [1] from line 144.
> > There is a long list of dependencies with `test` scope.
> > Actually, all of them are available as a transitive dependency from `spark-core`.
> >
> > We doesn't have to enlist them in every other project that doesn't use `flatten-plugin`.
> >
> > [1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> >
> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> > > Nikolay,
> > >
> > >
> > > Can you describe your problem and desired behaviour more thoroughly, please?
> > >
> > >
> > >
> > > > On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]> wrote:
> > > >
> > > > Hello, Igniters.
> > > >
> > > > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> > > > As far as I can understand it minimize pom.xml before it installed in repository.
> > > >
> > > > It introduce some strange behavior in modules:
> > > >
> > > > I must to enlist all test dependencies in my module to get tests work.
> > > > Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
> > > > It happens in `spark` [2] and `spark_2.10` [3] modules.
> > > > Now, when I want to enable testing of Spark Examples I has to enlist same dependencies in `examples/pom.xml`
> > > >
> > > > It looks like a mess for me.
> > > >
> > > > Please, help me:
> > > >
> > > > 1. Am I miss something and can make pom.xml much clearer?
> > > >
> > > > 2. Why we need to minimize pom.xml? It looks like other Apache project doesn't do it [5].
> > > >
> > > > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
> > > > [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> > > > [3] https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
> > > > [4] https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
> > > > [5] http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
> > >
> > >

[Andrey Novikov-3]

Nikolay,

I think it can be removed, if parent-pom will be installed(deployed).

On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]> wrote:

> Hello, Andrey.
>
> Thanks for an answer.
>
>> As I remember we use `flatten-maven-plugin` for flattening and removing parent relationship in deployed artifacts
>
> So we need it only in `release` profile?
>
> I found some earlier discussion about plugin [1]
>
>> in first versions of build, version was stored in variable in parent pom
>
> Got it. Do we need this plugin now?
>
> [1] http://apache-ignite-developers.2346864.n4.nabble.com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>
> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>> Nikolay,
>>
>> As I remember we use `flatten-maven-plugin` for flattening and
>> removing parent relationship in deployed artifacts (parent pom does
>> not deploy to repository and in first versions of build, version was
>> stored in variable in parent pom)
>>
>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]> wrote:
>> > Hello, Petr.
>> >
>> > > Can you describe your problem
>> >
>> > In Ignite, maven doesn't include transitive dependencies to test classpath.
>> > So we have enlist all dependencies in project pom.xml.
>> >
>> > > desired behaviour
>> >
>> > All I want is default maven behavior.
>> >
>> > Please, take a look at pom.xml [1] from line 144.
>> > There is a long list of dependencies with `test` scope.
>> > Actually, all of them are available as a transitive dependency from `spark-core`.
>> >
>> > We doesn't have to enlist them in every other project that doesn't use `flatten-plugin`.
>> >
>> > [1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>> >
>> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> > > Nikolay,
>> > >
>> > >
>> > > Can you describe your problem and desired behaviour more thoroughly, please?
>> > >
>> > >
>> > >
>> > > > On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]> wrote:
>> > > >
>> > > > Hello, Igniters.
>> > > >
>> > > > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> > > > As far as I can understand it minimize pom.xml before it installed in repository.
>> > > >
>> > > > It introduce some strange behavior in modules:
>> > > >
>> > > > I must to enlist all test dependencies in my module to get tests work.
>> > > > Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
>> > > > It happens in `spark` [2] and `spark_2.10` [3] modules.
>> > > > Now, when I want to enable testing of Spark Examples I has to enlist same dependencies in `examples/pom.xml`
>> > > >
>> > > > It looks like a mess for me.
>> > > >
>> > > > Please, help me:
>> > > >
>> > > > 1. Am I miss something and can make pom.xml much clearer?
>> > > >
>> > > > 2. Why we need to minimize pom.xml? It looks like other Apache project doesn't do it [5].
>> > > >
>> > > > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
>> > > > [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>> > > > [3] https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
>> > > > [4] https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
>> > > > [5] http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
>> > >
>> > >

[Nikolay Izhikov-2]

Andrey, thank you.

If there are no other objections I will create ticket for pom structure
refactoring.

Anton, can you help me with it as an Ignite Release Manager?

2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:

> Nikolay,
>
> I think it can be removed, if parent-pom will be installed(deployed).
>
> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]>
> wrote:
> > Hello, Andrey.
> >
> > Thanks for an answer.
> >
> >> As I remember we use `flatten-maven-plugin` for flattening and removing
> parent relationship in deployed artifacts
> >
> > So we need it only in `release` profile?
> >
> > I found some earlier discussion about plugin [1]
> >
> >> in first versions of build, version was stored in variable in parent pom
> >
> > Got it. Do we need this plugin now?
> >
> > [1] http://apache-ignite-developers.2346864.n4.nabble.
> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >
> > В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> >> Nikolay,
> >>
> >> As I remember we use `flatten-maven-plugin` for flattening and
> >> removing parent relationship in deployed artifacts (parent pom does
> >> not deploy to repository and in first versions of build, version was
> >> stored in variable in parent pom)
> >>
> >> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]>
> wrote:
> >> > Hello, Petr.
> >> >
> >> > > Can you describe your problem
> >> >
> >> > In Ignite, maven doesn't include transitive dependencies to test
> classpath.
> >> > So we have enlist all dependencies in project pom.xml.
> >> >
> >> > > desired behaviour
> >> >
> >> > All I want is default maven behavior.
> >> >
> >> > Please, take a look at pom.xml [1] from line 144.
> >> > There is a long list of dependencies with `test` scope.
> >> > Actually, all of them are available as a transitive dependency from
> `spark-core`.
> >> >
> >> > We doesn't have to enlist them in every other project that doesn't
> use `flatten-plugin`.
> >> >
> >> > [1] https://github.com/apache/ignite/blob/master/modules/
> spark/pom.xml#L144
> >> >
> >> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >> > > Nikolay,
> >> > >
> >> > >
> >> > > Can you describe your problem and desired behaviour more
> thoroughly, please?
> >> > >
> >> > >
> >> > >
> >> > > > On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
> wrote:
> >> > > >
> >> > > > Hello, Igniters.
> >> > > >
> >> > > > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >> > > > As far as I can understand it minimize pom.xml before it
> installed in repository.
> >> > > >
> >> > > > It introduce some strange behavior in modules:
> >> > > >
> >> > > > I must to enlist all test dependencies in my module to get tests
> work.
> >> > > > Otherwise tests fails with ClassNoDefFoundError for transitive
> dependencies.
> >> > > > It happens in `spark` [2] and `spark_2.10` [3] modules.
> >> > > > Now, when I want to enable testing of Spark Examples I has to
> enlist same dependencies in `examples/pom.xml`
> >> > > >
> >> > > > It looks like a mess for me.
> >> > > >
> >> > > > Please, help me:
> >> > > >
> >> > > > 1. Am I miss something and can make pom.xml much clearer?
> >> > > >
> >> > > > 2. Why we need to minimize pom.xml? It looks like other Apache
> project doesn't do it [5].
> >> > > >
> >> > > > [1] https://github.com/apache/ignite/blob/master/parent/pom.
> xml#L612
> >> > > > [2] https://github.com/apache/ignite/blob/master/modules/
> spark/pom.xml#L144
> >> > > > [3] https://github.com/apache/ignite/blob/master/modules/
> spark-2.10/pom.xml#L150
> >> > > > [4] https://github.com/apache/ignite/pull/3590/files#diff-
> 08740066c64337d38cccd84991ac0912R155
> >> > > > [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
> 2.12/1.0.0/kafka_2.12-1.0.0.pom
> >> > >
> >> > >
>

[vveider]

May be it will be enough to add a profile to parent pom.xml which activates on deploy stage only.

Yet, I still do not get a mechanism that causes a problem. What pom.xml becomes flattened, what is removed from it and how it effects other modules of the same project?

> On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
>
> Andrey, thank you.
>
> If there are no other objections I will create ticket for pom structure
> refactoring.
>
> Anton, can you help me with it as an Ignite Release Manager?
>
> 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
>
>> Nikolay,
>>
>> I think it can be removed, if parent-pom will be installed(deployed).
>>
>> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]>
>> wrote:
>>> Hello, Andrey.
>>>
>>> Thanks for an answer.
>>>
>>>> As I remember we use `flatten-maven-plugin` for flattening and removing
>> parent relationship in deployed artifacts
>>>
>>> So we need it only in `release` profile?
>>>
>>> I found some earlier discussion about plugin [1]
>>>
>>>> in first versions of build, version was stored in variable in parent pom
>>>
>>> Got it. Do we need this plugin now?
>>>
>>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>>>
>>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>>>> Nikolay,
>>>>
>>>> As I remember we use `flatten-maven-plugin` for flattening and
>>>> removing parent relationship in deployed artifacts (parent pom does
>>>> not deploy to repository and in first versions of build, version was
>>>> stored in variable in parent pom)
>>>>
>>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]>
>> wrote:
>>>>> Hello, Petr.
>>>>>
>>>>>> Can you describe your problem
>>>>>
>>>>> In Ignite, maven doesn't include transitive dependencies to test
>> classpath.
>>>>> So we have enlist all dependencies in project pom.xml.
>>>>>
>>>>>> desired behaviour
>>>>>
>>>>> All I want is default maven behavior.
>>>>>
>>>>> Please, take a look at pom.xml [1] from line 144.
>>>>> There is a long list of dependencies with `test` scope.
>>>>> Actually, all of them are available as a transitive dependency from
>> `spark-core`.
>>>>>
>>>>> We doesn't have to enlist them in every other project that doesn't
>> use `flatten-plugin`.
>>>>>
>>>>> [1] https://github.com/apache/ignite/blob/master/modules/
>> spark/pom.xml#L144
>>>>>
>>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>>>>>> Nikolay,
>>>>>>
>>>>>>
>>>>>> Can you describe your problem and desired behaviour more
>> thoroughly, please?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
>> wrote:
>>>>>>>
>>>>>>> Hello, Igniters.
>>>>>>>
>>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>>>>>>> As far as I can understand it minimize pom.xml before it
>> installed in repository.
>>>>>>>
>>>>>>> It introduce some strange behavior in modules:
>>>>>>>
>>>>>>> I must to enlist all test dependencies in my module to get tests
>> work.
>>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
>> dependencies.
>>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
>>>>>>> Now, when I want to enable testing of Spark Examples I has to
>> enlist same dependencies in `examples/pom.xml`
>>>>>>>
>>>>>>> It looks like a mess for me.
>>>>>>>
>>>>>>> Please, help me:
>>>>>>>
>>>>>>> 1. Am I miss something and can make pom.xml much clearer?
>>>>>>>
>>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
>> project doesn't do it [5].
>>>>>>>
>>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
>> xml#L612
>>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
>> spark/pom.xml#L144
>>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
>> spark-2.10/pom.xml#L150
>>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
>> 08740066c64337d38cccd84991ac0912R155
>>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
>> 2.12/1.0.0/kafka_2.12-1.0.0.pom
>>>>>>
>>>>>>
>>

[Anton Vinogradov-2]

 Nikolay,

Sure, I think we have to create IEP "Assembly and Release cleanup and
simplification".

I see some issues can be attached to this IEP
- Fabric word removal
- Flatten plugin removal (in possible)
- Ignite should be assemblied using package phase, not install in case you
want only to assembly it.
- Obsolete/unused/odd profiles removal
- scala 2.10 modules removal (if possible)
- Assembly simplification (get rid of ant scripts inside pom)
- Version update should be presented as sh script instead
of update-versions profile
- TeamCity repease procedure should be cleaned up (inc. RPM logic)
- etc.

2018-03-01 14:18 GMT+03:00 Petr Ivanov <[hidden email]>:

> May be it will be enough to add a profile to parent pom.xml which
> activates on deploy stage only.
>
> Yet, I still do not get a mechanism that causes a problem. What pom.xml
> becomes flattened, what is removed from it and how it effects other modules
> of the same project?
>
>
>
> > On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Andrey, thank you.
> >
> > If there are no other objections I will create ticket for pom structure
> > refactoring.
> >
> > Anton, can you help me with it as an Ignite Release Manager?
> >
> > 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
> >
> >> Nikolay,
> >>
> >> I think it can be removed, if parent-pom will be installed(deployed).
> >>
> >> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>> Hello, Andrey.
> >>>
> >>> Thanks for an answer.
> >>>
> >>>> As I remember we use `flatten-maven-plugin` for flattening and
> removing
> >> parent relationship in deployed artifacts
> >>>
> >>> So we need it only in `release` profile?
> >>>
> >>> I found some earlier discussion about plugin [1]
> >>>
> >>>> in first versions of build, version was stored in variable in parent
> pom
> >>>
> >>> Got it. Do we need this plugin now?
> >>>
> >>> [1] http://apache-ignite-developers.2346864.n4.nabble.
> >> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >>>
> >>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> >>>> Nikolay,
> >>>>
> >>>> As I remember we use `flatten-maven-plugin` for flattening and
> >>>> removing parent relationship in deployed artifacts (parent pom does
> >>>> not deploy to repository and in first versions of build, version was
> >>>> stored in variable in parent pom)
> >>>>
> >>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>>>> Hello, Petr.
> >>>>>
> >>>>>> Can you describe your problem
> >>>>>
> >>>>> In Ignite, maven doesn't include transitive dependencies to test
> >> classpath.
> >>>>> So we have enlist all dependencies in project pom.xml.
> >>>>>
> >>>>>> desired behaviour
> >>>>>
> >>>>> All I want is default maven behavior.
> >>>>>
> >>>>> Please, take a look at pom.xml [1] from line 144.
> >>>>> There is a long list of dependencies with `test` scope.
> >>>>> Actually, all of them are available as a transitive dependency from
> >> `spark-core`.
> >>>>>
> >>>>> We doesn't have to enlist them in every other project that doesn't
> >> use `flatten-plugin`.
> >>>>>
> >>>>> [1] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >>>>>
> >>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >>>>>> Nikolay,
> >>>>>>
> >>>>>>
> >>>>>> Can you describe your problem and desired behaviour more
> >> thoroughly, please?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>>>>>>
> >>>>>>> Hello, Igniters.
> >>>>>>>
> >>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >>>>>>> As far as I can understand it minimize pom.xml before it
> >> installed in repository.
> >>>>>>>
> >>>>>>> It introduce some strange behavior in modules:
> >>>>>>>
> >>>>>>> I must to enlist all test dependencies in my module to get tests
> >> work.
> >>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
> >> dependencies.
> >>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
> >>>>>>> Now, when I want to enable testing of Spark Examples I has to
> >> enlist same dependencies in `examples/pom.xml`
> >>>>>>>
> >>>>>>> It looks like a mess for me.
> >>>>>>>
> >>>>>>> Please, help me:
> >>>>>>>
> >>>>>>> 1. Am I miss something and can make pom.xml much clearer?
> >>>>>>>
> >>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
> >> project doesn't do it [5].
> >>>>>>>
> >>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
> >> xml#L612
> >>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
> >> spark-2.10/pom.xml#L150
> >>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
> >> 08740066c64337d38cccd84991ac0912R155
> >>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
> >> 2.12/1.0.0/kafka_2.12-1.0.0.pom
> >>>>>>
> >>>>>>
> >>
>
>

[Nikolay Izhikov-2]

> May be it will be enough to add a profile to parent pom.xml which
activates on deploy stage only.

I think it will be enough to solve my issue.

> What pom.xml becomes flattened, what is removed from it and how it effects
other modules of the same project?

pom.xml that are installed to local repository is flattened.

> Yet, I still do not get a mechanism that causes a problem.

If you want to reproduce it, just do the following steps:

1. remove spark dependencies with `scope=test` from `spark/pom.xml`

2. run:

`~/src/ignite/modules/spark> mvn install -U
-Plgpl,examples,scala,-clean-libs,-release,ml
-Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
-Dmaven.javadoc.skip=true -DfailIfNoTests=false`

It will execute OK, because maven uses pom.xml from source.

3. run(only root dir is changed)

`~/src/ignite/> mvn install -U
-Plgpl,examples,scala,-clean-libs,-release,ml
-Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
-Dmaven.javadoc.skip=true -DfailIfNoTests=false`

Test will fail with ClassNoDefFoundError because maven will use
pom.xml from local repository which is flattened.

4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
Retry step 3. It will succeed.


2018-03-01 14:18 GMT+03:00 Petr Ivanov <[hidden email]>:

> May be it will be enough to add a profile to parent pom.xml which
> activates on deploy stage only.
>
> Yet, I still do not get a mechanism that causes a problem. What pom.xml
> becomes flattened, what is removed from it and how it effects other modules
> of the same project?
>
>
>
> > On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Andrey, thank you.
> >
> > If there are no other objections I will create ticket for pom structure
> > refactoring.
> >
> > Anton, can you help me with it as an Ignite Release Manager?
> >
> > 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
> >
> >> Nikolay,
> >>
> >> I think it can be removed, if parent-pom will be installed(deployed).
> >>
> >> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>> Hello, Andrey.
> >>>
> >>> Thanks for an answer.
> >>>
> >>>> As I remember we use `flatten-maven-plugin` for flattening and
> removing
> >> parent relationship in deployed artifacts
> >>>
> >>> So we need it only in `release` profile?
> >>>
> >>> I found some earlier discussion about plugin [1]
> >>>
> >>>> in first versions of build, version was stored in variable in parent
> pom
> >>>
> >>> Got it. Do we need this plugin now?
> >>>
> >>> [1] http://apache-ignite-developers.2346864.n4.nabble.
> >> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >>>
> >>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> >>>> Nikolay,
> >>>>
> >>>> As I remember we use `flatten-maven-plugin` for flattening and
> >>>> removing parent relationship in deployed artifacts (parent pom does
> >>>> not deploy to repository and in first versions of build, version was
> >>>> stored in variable in parent pom)
> >>>>
> >>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>>>> Hello, Petr.
> >>>>>
> >>>>>> Can you describe your problem
> >>>>>
> >>>>> In Ignite, maven doesn't include transitive dependencies to test
> >> classpath.
> >>>>> So we have enlist all dependencies in project pom.xml.
> >>>>>
> >>>>>> desired behaviour
> >>>>>
> >>>>> All I want is default maven behavior.
> >>>>>
> >>>>> Please, take a look at pom.xml [1] from line 144.
> >>>>> There is a long list of dependencies with `test` scope.
> >>>>> Actually, all of them are available as a transitive dependency from
> >> `spark-core`.
> >>>>>
> >>>>> We doesn't have to enlist them in every other project that doesn't
> >> use `flatten-plugin`.
> >>>>>
> >>>>> [1] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >>>>>
> >>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >>>>>> Nikolay,
> >>>>>>
> >>>>>>
> >>>>>> Can you describe your problem and desired behaviour more
> >> thoroughly, please?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>>>>>>
> >>>>>>> Hello, Igniters.
> >>>>>>>
> >>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >>>>>>> As far as I can understand it minimize pom.xml before it
> >> installed in repository.
> >>>>>>>
> >>>>>>> It introduce some strange behavior in modules:
> >>>>>>>
> >>>>>>> I must to enlist all test dependencies in my module to get tests
> >> work.
> >>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
> >> dependencies.
> >>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
> >>>>>>> Now, when I want to enable testing of Spark Examples I has to
> >> enlist same dependencies in `examples/pom.xml`
> >>>>>>>
> >>>>>>> It looks like a mess for me.
> >>>>>>>
> >>>>>>> Please, help me:
> >>>>>>>
> >>>>>>> 1. Am I miss something and can make pom.xml much clearer?
> >>>>>>>
> >>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
> >> project doesn't do it [5].
> >>>>>>>
> >>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
> >> xml#L612
> >>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
> >> spark-2.10/pom.xml#L150
> >>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
> >> 08740066c64337d38cccd84991ac0912R155
> >>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
> >> 2.12/1.0.0/kafka_2.12-1.0.0.pom
> >>>>>>
> >>>>>>
> >>
>
>

[daradurvs]

I agree with Anton, especially with the following statement:
>> Ignite should be assemblied using package phase,
>> not install in case you want only to assembly it.

Approach with "install" phase may lead to hidden issues in a
development environment.

On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov <[hidden email]> wrote:

>> May be it will be enough to add a profile to parent pom.xml which
> activates on deploy stage only.
>
> I think it will be enough to solve my issue.
>
>> What pom.xml becomes flattened, what is removed from it and how it effects
> other modules of the same project?
>
> pom.xml that are installed to local repository is flattened.
>
>> Yet, I still do not get a mechanism that causes a problem.
>
> If you want to reproduce it, just do the following steps:
>
> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
>
> 2. run:
>
> `~/src/ignite/modules/spark> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> It will execute OK, because maven uses pom.xml from source.
>
> 3. run(only root dir is changed)
>
> `~/src/ignite/> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> Test will fail with ClassNoDefFoundError because maven will use
> pom.xml from local repository which is flattened.
>
> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
> Retry step 3. It will succeed.
>
>
> 2018-03-01 14:18 GMT+03:00 Petr Ivanov <[hidden email]>:
>
>> May be it will be enough to add a profile to parent pom.xml which
>> activates on deploy stage only.
>>
>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
>> becomes flattened, what is removed from it and how it effects other modules
>> of the same project?
>>
>>
>>
>> > On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
>> >
>> > Andrey, thank you.
>> >
>> > If there are no other objections I will create ticket for pom structure
>> > refactoring.
>> >
>> > Anton, can you help me with it as an Ignite Release Manager?
>> >
>> > 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
>> >
>> >> Nikolay,
>> >>
>> >> I think it can be removed, if parent-pom will be installed(deployed).
>> >>
>> >> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]>
>> >> wrote:
>> >>> Hello, Andrey.
>> >>>
>> >>> Thanks for an answer.
>> >>>
>> >>>> As I remember we use `flatten-maven-plugin` for flattening and
>> removing
>> >> parent relationship in deployed artifacts
>> >>>
>> >>> So we need it only in `release` profile?
>> >>>
>> >>> I found some earlier discussion about plugin [1]
>> >>>
>> >>>> in first versions of build, version was stored in variable in parent
>> pom
>> >>>
>> >>> Got it. Do we need this plugin now?
>> >>>
>> >>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> >> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>> >>>
>> >>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>> >>>> Nikolay,
>> >>>>
>> >>>> As I remember we use `flatten-maven-plugin` for flattening and
>> >>>> removing parent relationship in deployed artifacts (parent pom does
>> >>>> not deploy to repository and in first versions of build, version was
>> >>>> stored in variable in parent pom)
>> >>>>
>> >>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]>
>> >> wrote:
>> >>>>> Hello, Petr.
>> >>>>>
>> >>>>>> Can you describe your problem
>> >>>>>
>> >>>>> In Ignite, maven doesn't include transitive dependencies to test
>> >> classpath.
>> >>>>> So we have enlist all dependencies in project pom.xml.
>> >>>>>
>> >>>>>> desired behaviour
>> >>>>>
>> >>>>> All I want is default maven behavior.
>> >>>>>
>> >>>>> Please, take a look at pom.xml [1] from line 144.
>> >>>>> There is a long list of dependencies with `test` scope.
>> >>>>> Actually, all of them are available as a transitive dependency from
>> >> `spark-core`.
>> >>>>>
>> >>>>> We doesn't have to enlist them in every other project that doesn't
>> >> use `flatten-plugin`.
>> >>>>>
>> >>>>> [1] https://github.com/apache/ignite/blob/master/modules/
>> >> spark/pom.xml#L144
>> >>>>>
>> >>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> >>>>>> Nikolay,
>> >>>>>>
>> >>>>>>
>> >>>>>> Can you describe your problem and desired behaviour more
>> >> thoroughly, please?
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
>> >> wrote:
>> >>>>>>>
>> >>>>>>> Hello, Igniters.
>> >>>>>>>
>> >>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> >>>>>>> As far as I can understand it minimize pom.xml before it
>> >> installed in repository.
>> >>>>>>>
>> >>>>>>> It introduce some strange behavior in modules:
>> >>>>>>>
>> >>>>>>> I must to enlist all test dependencies in my module to get tests
>> >> work.
>> >>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
>> >> dependencies.
>> >>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
>> >>>>>>> Now, when I want to enable testing of Spark Examples I has to
>> >> enlist same dependencies in `examples/pom.xml`
>> >>>>>>>
>> >>>>>>> It looks like a mess for me.
>> >>>>>>>
>> >>>>>>> Please, help me:
>> >>>>>>>
>> >>>>>>> 1. Am I miss something and can make pom.xml much clearer?
>> >>>>>>>
>> >>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
>> >> project doesn't do it [5].
>> >>>>>>>
>> >>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
>> >> xml#L612
>> >>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
>> >> spark/pom.xml#L144
>> >>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
>> >> spark-2.10/pom.xml#L150
>> >>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
>> >> 08740066c64337d38cccd84991ac0912R155
>> >>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
>> >> 2.12/1.0.0/kafka_2.12-1.0.0.pom
>> >>>>>>
>> >>>>>>
>> >>
>>
>>

--
Best Regards, Vyacheslav D.

[vveider]

> 2. run:
>
> `~/src/ignite/modules/spark> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> It will execute OK, because maven uses pom.xml from source.

Which pom.xml? From the module modules/spark?
Also there is no profiles lgpl,examples,scala,-clean-libs,-release,ml in that module.

>
> 3. run(only root dir is changed)
>
> `~/src/ignite/> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`

As I understand maven, test will be executed using dependencies from the reactor, not from local repository.


What am I missing?

[vveider]

Anton, Vyacheslav — are you proposing assembly process shift to package phase, yet leaving the special profile for assembly itself?

> On 1 Mar 2018, at 14:36, Vyacheslav Daradur <[hidden email]> wrote:
>
> I agree with Anton, especially with the following statement:
>>> Ignite should be assemblied using package phase,
>>> not install in case you want only to assembly it.
>
> Approach with "install" phase may lead to hidden issues in a
> development environment.
>
> On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov <[hidden email]> wrote:
>>> May be it will be enough to add a profile to parent pom.xml which
>> activates on deploy stage only.
>>
>> I think it will be enough to solve my issue.
>>
>>> What pom.xml becomes flattened, what is removed from it and how it effects
>> other modules of the same project?
>>
>> pom.xml that are installed to local repository is flattened.
>>
>>> Yet, I still do not get a mechanism that causes a problem.
>>
>> If you want to reproduce it, just do the following steps:
>>
>> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
>>
>> 2. run:
>>
>> `~/src/ignite/modules/spark> mvn install -U
>> -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>>
>> It will execute OK, because maven uses pom.xml from source.
>>
>> 3. run(only root dir is changed)
>>
>> `~/src/ignite/> mvn install -U
>> -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>>
>> Test will fail with ClassNoDefFoundError because maven will use
>> pom.xml from local repository which is flattened.
>>
>> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
>> Retry step 3. It will succeed.
>>
>>
>> 2018-03-01 14:18 GMT+03:00 Petr Ivanov <[hidden email]>:
>>
>>> May be it will be enough to add a profile to parent pom.xml which
>>> activates on deploy stage only.
>>>
>>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
>>> becomes flattened, what is removed from it and how it effects other modules
>>> of the same project?
>>>
>>>
>>>
>>>> On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
>>>>
>>>> Andrey, thank you.
>>>>
>>>> If there are no other objections I will create ticket for pom structure
>>>> refactoring.
>>>>
>>>> Anton, can you help me with it as an Ignite Release Manager?
>>>>
>>>> 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
>>>>
>>>>> Nikolay,
>>>>>
>>>>> I think it can be removed, if parent-pom will be installed(deployed).
>>>>>
>>>>> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]>
>>>>> wrote:
>>>>>> Hello, Andrey.
>>>>>>
>>>>>> Thanks for an answer.
>>>>>>
>>>>>>> As I remember we use `flatten-maven-plugin` for flattening and
>>> removing
>>>>> parent relationship in deployed artifacts
>>>>>>
>>>>>> So we need it only in `release` profile?
>>>>>>
>>>>>> I found some earlier discussion about plugin [1]
>>>>>>
>>>>>>> in first versions of build, version was stored in variable in parent
>>> pom
>>>>>>
>>>>>> Got it. Do we need this plugin now?
>>>>>>
>>>>>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>>>>> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>>>>>>
>>>>>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>>>>>>> Nikolay,
>>>>>>>
>>>>>>> As I remember we use `flatten-maven-plugin` for flattening and
>>>>>>> removing parent relationship in deployed artifacts (parent pom does
>>>>>>> not deploy to repository and in first versions of build, version was
>>>>>>> stored in variable in parent pom)
>>>>>>>
>>>>>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <[hidden email]>
>>>>> wrote:
>>>>>>>> Hello, Petr.
>>>>>>>>
>>>>>>>>> Can you describe your problem
>>>>>>>>
>>>>>>>> In Ignite, maven doesn't include transitive dependencies to test
>>>>> classpath.
>>>>>>>> So we have enlist all dependencies in project pom.xml.
>>>>>>>>
>>>>>>>>> desired behaviour
>>>>>>>>
>>>>>>>> All I want is default maven behavior.
>>>>>>>>
>>>>>>>> Please, take a look at pom.xml [1] from line 144.
>>>>>>>> There is a long list of dependencies with `test` scope.
>>>>>>>> Actually, all of them are available as a transitive dependency from
>>>>> `spark-core`.
>>>>>>>>
>>>>>>>> We doesn't have to enlist them in every other project that doesn't
>>>>> use `flatten-plugin`.
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/ignite/blob/master/modules/
>>>>> spark/pom.xml#L144
>>>>>>>>
>>>>>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>>>>>>>>> Nikolay,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you describe your problem and desired behaviour more
>>>>> thoroughly, please?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello, Igniters.
>>>>>>>>>>
>>>>>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>>>>>>>>>> As far as I can understand it minimize pom.xml before it
>>>>> installed in repository.
>>>>>>>>>>
>>>>>>>>>> It introduce some strange behavior in modules:
>>>>>>>>>>
>>>>>>>>>> I must to enlist all test dependencies in my module to get tests
>>>>> work.
>>>>>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
>>>>> dependencies.
>>>>>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
>>>>>>>>>> Now, when I want to enable testing of Spark Examples I has to
>>>>> enlist same dependencies in `examples/pom.xml`
>>>>>>>>>>
>>>>>>>>>> It looks like a mess for me.
>>>>>>>>>>
>>>>>>>>>> Please, help me:
>>>>>>>>>>
>>>>>>>>>> 1. Am I miss something and can make pom.xml much clearer?
>>>>>>>>>>
>>>>>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
>>>>> project doesn't do it [5].
>>>>>>>>>>
>>>>>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
>>>>> xml#L612
>>>>>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
>>>>> spark/pom.xml#L144
>>>>>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
>>>>> spark-2.10/pom.xml#L150
>>>>>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
>>>>> 08740066c64337d38cccd84991ac0912R155
>>>>>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
>>>>> 2.12/1.0.0/kafka_2.12-1.0.0.pom
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>
>>>
>
>
>
> --
> Best Regards, Vyacheslav D.

[Nikolay Izhikov-2]

> Which pom.xml? From the module modules/spark?

Yes.

> As I understand maven, test will be executed using dependencies from the
reactor, not from local repository.

I don't get what is the point.
Did you try to reproduce issue?
Or should I provide full traces to you?

> What am I missing?

I see that whole build process is done in a very strange way.
Because of flatten plugin.

I described and able reproduce(both locally and on TC) specific issues that
turns whole pom.xml into the mess.
I'm pretty sure that source of issue is flatten plugin.
I'm not the first person who got this errors.

My propositions are:

1. Let's discuss how we should fix it.
2. Let's fix it!


2018-03-01 15:35 GMT+03:00 Petr Ivanov <[hidden email]>:

>
> > 2. run:
> >
> > `~/src/ignite/modules/spark> mvn install -U
> > -Plgpl,examples,scala,-clean-libs,-release,ml
> > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> > -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> >
> > It will execute OK, because maven uses pom.xml from source.
>
> Which pom.xml? From the module modules/spark?
> Also there is no profiles lgpl,examples,scala,-clean-libs,-release,ml in
> that module.
>
> >
> > 3. run(only root dir is changed)
> >
> > `~/src/ignite/> mvn install -U
> > -Plgpl,examples,scala,-clean-libs,-release,ml
> > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> > -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> As I understand maven, test will be executed using dependencies from the
> reactor, not from local repository.
>
>
> What am I missing?

[vveider]

>
> I don't get what is the point.
> Did you try to reproduce issue?
> Or should I provide full traces to you?
>

My point in inability to fully understand and describe the problem in terms of mechanism which causes it.
For now I can see only indirect guessing.


And yes, I’ve run your commands and both times (in spark module directory and in root) it produces the same result:

1.
ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite -Dmaven.javadoc.skip=true -DfailIfNoTests=false
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
[WARNING] The requested profile "ml" could not be activated because it does not exist.

2.
ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite -Dmaven.javadoc.skip=true -DfailIfNoTests=false
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
[WARNING] The requested profile "lgpl" could not be activated because it does not exist.
[WARNING] The requested profile "examples" could not be activated because it does not exist.
[WARNING] The requested profile "scala" could not be activated because it does not exist.
[WARNING] The requested profile "ml" could not be activated because it does not exist.

[Nikolay Izhikov-2]

Petr.

Thank you for trying!

Did you remove 'test' dependencies before running commands?

Because I commit in master only correct pom.xml for current build process,
of course.

But, to make things works I have to copy paste transitive dependencies from
spark-core.

1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" <[hidden email]>
написал:

>
> >
> > I don't get what is the point.
> > Did you try to reproduce issue?
> > Or should I provide full traces to you?
> >
>
> My point in inability to fully understand and describe the problem in
> terms of mechanism which causes it.
> For now I can see only indirect guessing.
>
>
> And yes, I’ve run your commands and both times (in spark module directory
> and in root) it produces the same result:
>
> 1.
> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn
> install -U -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
> [WARNING] The requested profile "ml" could not be activated because it
> does not exist.
>
> 2.
> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v
> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
> [WARNING] The requested profile "lgpl" could not be activated because it
> does not exist.
> [WARNING] The requested profile "examples" could not be activated because
> it does not exist.
> [WARNING] The requested profile "scala" could not be activated because it
> does not exist.
> [WARNING] The requested profile "ml" could not be activated because it
> does not exist.

[Dmitriy Pavlov]

Hi,

I think we should solve initial problem: "We have to include all transitive
dependencies into poms". I've faced with it during Direct IO plugin
development.

I am sure it is not best practice. Because 1st level dependency version
change will require us to update 2nd level dependencies manually.

So if we can find solution without disabling plugin, we should prefer it.
If only disable may help we should be sure release installation into
maven-central will be successful.

Sincerely,
Dmitriy Pavlov

чт, 1 мар. 2018 г. в 17:23, Petr Ivanov <[hidden email]>:

> Anton, Vyacheslav — are you proposing assembly process shift to package
> phase, yet leaving the special profile for assembly itself?
>
>
>
> > On 1 Mar 2018, at 14:36, Vyacheslav Daradur <[hidden email]> wrote:
> >
> > I agree with Anton, especially with the following statement:
> >>> Ignite should be assemblied using package phase,
> >>> not install in case you want only to assembly it.
> >
> > Approach with "install" phase may lead to hidden issues in a
> > development environment.
> >
> > On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov <[hidden email]>
> wrote:
> >>> May be it will be enough to add a profile to parent pom.xml which
> >> activates on deploy stage only.
> >>
> >> I think it will be enough to solve my issue.
> >>
> >>> What pom.xml becomes flattened, what is removed from it and how it
> effects
> >> other modules of the same project?
> >>
> >> pom.xml that are installed to local repository is flattened.
> >>
> >>> Yet, I still do not get a mechanism that causes a problem.
> >>
> >> If you want to reproduce it, just do the following steps:
> >>
> >> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
> >>
> >> 2. run:
> >>
> >> `~/src/ignite/modules/spark> mvn install -U
> >> -Plgpl,examples,scala,-clean-libs,-release,ml
> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> >>
> >> It will execute OK, because maven uses pom.xml from source.
> >>
> >> 3. run(only root dir is changed)
> >>
> >> `~/src/ignite/> mvn install -U
> >> -Plgpl,examples,scala,-clean-libs,-release,ml
> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> >>
> >> Test will fail with ClassNoDefFoundError because maven will use
> >> pom.xml from local repository which is flattened.
> >>
> >> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
> >> Retry step 3. It will succeed.
> >>
> >>
> >> 2018-03-01 14:18 GMT+03:00 Petr Ivanov <[hidden email]>:
> >>
> >>> May be it will be enough to add a profile to parent pom.xml which
> >>> activates on deploy stage only.
> >>>
> >>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
> >>> becomes flattened, what is removed from it and how it effects other
> modules
> >>> of the same project?
> >>>
> >>>
> >>>
> >>>> On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
> >>>>
> >>>> Andrey, thank you.
> >>>>
> >>>> If there are no other objections I will create ticket for pom
> structure
> >>>> refactoring.
> >>>>
> >>>> Anton, can you help me with it as an Ignite Release Manager?
> >>>>
> >>>> 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
> >>>>
> >>>>> Nikolay,
> >>>>>
> >>>>> I think it can be removed, if parent-pom will be installed(deployed).
> >>>>>
> >>>>> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]
> >
> >>>>> wrote:
> >>>>>> Hello, Andrey.
> >>>>>>
> >>>>>> Thanks for an answer.
> >>>>>>
> >>>>>>> As I remember we use `flatten-maven-plugin` for flattening and
> >>> removing
> >>>>> parent relationship in deployed artifacts
> >>>>>>
> >>>>>> So we need it only in `release` profile?
> >>>>>>
> >>>>>> I found some earlier discussion about plugin [1]
> >>>>>>
> >>>>>>> in first versions of build, version was stored in variable in
> parent
> >>> pom
> >>>>>>
> >>>>>> Got it. Do we need this plugin now?
> >>>>>>
> >>>>>> [1] http://apache-ignite-developers.2346864.n4.nabble.
> >>>>> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >>>>>>
> >>>>>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> >>>>>>> Nikolay,
> >>>>>>>
> >>>>>>> As I remember we use `flatten-maven-plugin` for flattening and
> >>>>>>> removing parent relationship in deployed artifacts (parent pom does
> >>>>>>> not deploy to repository and in first versions of build, version
> was
> >>>>>>> stored in variable in parent pom)
> >>>>>>>
> >>>>>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <
> [hidden email]>
> >>>>> wrote:
> >>>>>>>> Hello, Petr.
> >>>>>>>>
> >>>>>>>>> Can you describe your problem
> >>>>>>>>
> >>>>>>>> In Ignite, maven doesn't include transitive dependencies to test
> >>>>> classpath.
> >>>>>>>> So we have enlist all dependencies in project pom.xml.
> >>>>>>>>
> >>>>>>>>> desired behaviour
> >>>>>>>>
> >>>>>>>> All I want is default maven behavior.
> >>>>>>>>
> >>>>>>>> Please, take a look at pom.xml [1] from line 144.
> >>>>>>>> There is a long list of dependencies with `test` scope.
> >>>>>>>> Actually, all of them are available as a transitive dependency
> from
> >>>>> `spark-core`.
> >>>>>>>>
> >>>>>>>> We doesn't have to enlist them in every other project that doesn't
> >>>>> use `flatten-plugin`.
> >>>>>>>>
> >>>>>>>> [1] https://github.com/apache/ignite/blob/master/modules/
> >>>>> spark/pom.xml#L144
> >>>>>>>>
> >>>>>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >>>>>>>>> Nikolay,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Can you describe your problem and desired behaviour more
> >>>>> thoroughly, please?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello, Igniters.
> >>>>>>>>>>
> >>>>>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >>>>>>>>>> As far as I can understand it minimize pom.xml before it
> >>>>> installed in repository.
> >>>>>>>>>>
> >>>>>>>>>> It introduce some strange behavior in modules:
> >>>>>>>>>>
> >>>>>>>>>> I must to enlist all test dependencies in my module to get tests
> >>>>> work.
> >>>>>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
> >>>>> dependencies.
> >>>>>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
> >>>>>>>>>> Now, when I want to enable testing of Spark Examples I has to
> >>>>> enlist same dependencies in `examples/pom.xml`
> >>>>>>>>>>
> >>>>>>>>>> It looks like a mess for me.
> >>>>>>>>>>
> >>>>>>>>>> Please, help me:
> >>>>>>>>>>
> >>>>>>>>>> 1. Am I miss something and can make pom.xml much clearer?
> >>>>>>>>>>
> >>>>>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
> >>>>> project doesn't do it [5].
> >>>>>>>>>>
> >>>>>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
> >>>>> xml#L612
> >>>>>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
> >>>>> spark/pom.xml#L144
> >>>>>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
> >>>>> spark-2.10/pom.xml#L150
> >>>>>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
> >>>>> 08740066c64337d38cccd84991ac0912R155
> >>>>>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
> >>>>> 2.12/1.0.0/kafka_2.12-1.0.0.pom
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>
> >>>
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>
>

[daradurvs]

>> Anton, Vyacheslav — are you proposing assembly process shift to package phase, yet leaving the special profile for assembly itself?

I'm talking about compile and package processes like it has been until
2.1 release.

On Thu, Mar 1, 2018 at 6:00 PM, Dmitry Pavlov <[hidden email]> wrote:

> Hi,
>
> I think we should solve initial problem: "We have to include all transitive
> dependencies into poms". I've faced with it during Direct IO plugin
> development.
>
> I am sure it is not best practice. Because 1st level dependency version
> change will require us to update 2nd level dependencies manually.
>
> So if we can find solution without disabling plugin, we should prefer it.
> If only disable may help we should be sure release installation into
> maven-central will be successful.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 1 мар. 2018 г. в 17:23, Petr Ivanov <[hidden email]>:
>
>> Anton, Vyacheslav — are you proposing assembly process shift to package
>> phase, yet leaving the special profile for assembly itself?
>>
>>
>>
>> > On 1 Mar 2018, at 14:36, Vyacheslav Daradur <[hidden email]> wrote:
>> >
>> > I agree with Anton, especially with the following statement:
>> >>> Ignite should be assemblied using package phase,
>> >>> not install in case you want only to assembly it.
>> >
>> > Approach with "install" phase may lead to hidden issues in a
>> > development environment.
>> >
>> > On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov <[hidden email]>
>> wrote:
>> >>> May be it will be enough to add a profile to parent pom.xml which
>> >> activates on deploy stage only.
>> >>
>> >> I think it will be enough to solve my issue.
>> >>
>> >>> What pom.xml becomes flattened, what is removed from it and how it
>> effects
>> >> other modules of the same project?
>> >>
>> >> pom.xml that are installed to local repository is flattened.
>> >>
>> >>> Yet, I still do not get a mechanism that causes a problem.
>> >>
>> >> If you want to reproduce it, just do the following steps:
>> >>
>> >> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
>> >>
>> >> 2. run:
>> >>
>> >> `~/src/ignite/modules/spark> mvn install -U
>> >> -Plgpl,examples,scala,-clean-libs,-release,ml
>> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>> >>
>> >> It will execute OK, because maven uses pom.xml from source.
>> >>
>> >> 3. run(only root dir is changed)
>> >>
>> >> `~/src/ignite/> mvn install -U
>> >> -Plgpl,examples,scala,-clean-libs,-release,ml
>> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>> >>
>> >> Test will fail with ClassNoDefFoundError because maven will use
>> >> pom.xml from local repository which is flattened.
>> >>
>> >> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
>> >> Retry step 3. It will succeed.
>> >>
>> >>
>> >> 2018-03-01 14:18 GMT+03:00 Petr Ivanov <[hidden email]>:
>> >>
>> >>> May be it will be enough to add a profile to parent pom.xml which
>> >>> activates on deploy stage only.
>> >>>
>> >>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
>> >>> becomes flattened, what is removed from it and how it effects other
>> modules
>> >>> of the same project?
>> >>>
>> >>>
>> >>>
>> >>>> On 1 Mar 2018, at 14:06, Nikolay Izhikov <[hidden email]> wrote:
>> >>>>
>> >>>> Andrey, thank you.
>> >>>>
>> >>>> If there are no other objections I will create ticket for pom
>> structure
>> >>>> refactoring.
>> >>>>
>> >>>> Anton, can you help me with it as an Ignite Release Manager?
>> >>>>
>> >>>> 2018-03-01 13:45 GMT+03:00 Andrey Novikov <[hidden email]>:
>> >>>>
>> >>>>> Nikolay,
>> >>>>>
>> >>>>> I think it can be removed, if parent-pom will be installed(deployed).
>> >>>>>
>> >>>>> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov <[hidden email]
>> >
>> >>>>> wrote:
>> >>>>>> Hello, Andrey.
>> >>>>>>
>> >>>>>> Thanks for an answer.
>> >>>>>>
>> >>>>>>> As I remember we use `flatten-maven-plugin` for flattening and
>> >>> removing
>> >>>>> parent relationship in deployed artifacts
>> >>>>>>
>> >>>>>> So we need it only in `release` profile?
>> >>>>>>
>> >>>>>> I found some earlier discussion about plugin [1]
>> >>>>>>
>> >>>>>>> in first versions of build, version was stored in variable in
>> parent
>> >>> pom
>> >>>>>>
>> >>>>>> Got it. Do we need this plugin now?
>> >>>>>>
>> >>>>>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> >>>>> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>> >>>>>>
>> >>>>>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>> >>>>>>> Nikolay,
>> >>>>>>>
>> >>>>>>> As I remember we use `flatten-maven-plugin` for flattening and
>> >>>>>>> removing parent relationship in deployed artifacts (parent pom does
>> >>>>>>> not deploy to repository and in first versions of build, version
>> was
>> >>>>>>> stored in variable in parent pom)
>> >>>>>>>
>> >>>>>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <
>> [hidden email]>
>> >>>>> wrote:
>> >>>>>>>> Hello, Petr.
>> >>>>>>>>
>> >>>>>>>>> Can you describe your problem
>> >>>>>>>>
>> >>>>>>>> In Ignite, maven doesn't include transitive dependencies to test
>> >>>>> classpath.
>> >>>>>>>> So we have enlist all dependencies in project pom.xml.
>> >>>>>>>>
>> >>>>>>>>> desired behaviour
>> >>>>>>>>
>> >>>>>>>> All I want is default maven behavior.
>> >>>>>>>>
>> >>>>>>>> Please, take a look at pom.xml [1] from line 144.
>> >>>>>>>> There is a long list of dependencies with `test` scope.
>> >>>>>>>> Actually, all of them are available as a transitive dependency
>> from
>> >>>>> `spark-core`.
>> >>>>>>>>
>> >>>>>>>> We doesn't have to enlist them in every other project that doesn't
>> >>>>> use `flatten-plugin`.
>> >>>>>>>>
>> >>>>>>>> [1] https://github.com/apache/ignite/blob/master/modules/
>> >>>>> spark/pom.xml#L144
>> >>>>>>>>
>> >>>>>>>> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> >>>>>>>>> Nikolay,
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Can you describe your problem and desired behaviour more
>> >>>>> thoroughly, please?
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov <[hidden email]>
>> >>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Hello, Igniters.
>> >>>>>>>>>>
>> >>>>>>>>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> >>>>>>>>>> As far as I can understand it minimize pom.xml before it
>> >>>>> installed in repository.
>> >>>>>>>>>>
>> >>>>>>>>>> It introduce some strange behavior in modules:
>> >>>>>>>>>>
>> >>>>>>>>>> I must to enlist all test dependencies in my module to get tests
>> >>>>> work.
>> >>>>>>>>>> Otherwise tests fails with ClassNoDefFoundError for transitive
>> >>>>> dependencies.
>> >>>>>>>>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
>> >>>>>>>>>> Now, when I want to enable testing of Spark Examples I has to
>> >>>>> enlist same dependencies in `examples/pom.xml`
>> >>>>>>>>>>
>> >>>>>>>>>> It looks like a mess for me.
>> >>>>>>>>>>
>> >>>>>>>>>> Please, help me:
>> >>>>>>>>>>
>> >>>>>>>>>> 1. Am I miss something and can make pom.xml much clearer?
>> >>>>>>>>>>
>> >>>>>>>>>> 2. Why we need to minimize pom.xml? It looks like other Apache
>> >>>>> project doesn't do it [5].
>> >>>>>>>>>>
>> >>>>>>>>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
>> >>>>> xml#L612
>> >>>>>>>>>> [2] https://github.com/apache/ignite/blob/master/modules/
>> >>>>> spark/pom.xml#L144
>> >>>>>>>>>> [3] https://github.com/apache/ignite/blob/master/modules/
>> >>>>> spark-2.10/pom.xml#L150
>> >>>>>>>>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
>> >>>>> 08740066c64337d38cccd84991ac0912R155
>> >>>>>>>>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
>> >>>>> 2.12/1.0.0/kafka_2.12-1.0.0.pom
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>
>> >>>
>> >>>
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>>
>>

--
Best Regards, Vyacheslav D.

[vveider]

The problem is solved by updating flatten-maven-plugin version to 1.0.1.

Nikolay, please, double check it.
If it really solves the problem, please, fill the ticket (or point to existing one), so I can update it and check impact on release procedure.

> On 1 Mar 2018, at 17:04, Nikolay Izhikov <[hidden email]> wrote:
>
> Petr.
>
> Thank you for trying!
>
> Did you remove 'test' dependencies before running commands?
>
> Because I commit in master only correct pom.xml for current build process,
> of course.
>
> But, to make things works I have to copy paste transitive dependencies from
> spark-core.
>
> 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" <[hidden email]>
> написал:
>
>>
>>>
>>> I don't get what is the point.
>>> Did you try to reproduce issue?
>>> Or should I provide full traces to you?
>>>
>>
>> My point in inability to fully understand and describe the problem in
>> terms of mechanism which causes it.
>> For now I can see only indirect guessing.
>>
>>
>> And yes, I’ve run your commands and both times (in spark module directory
>> and in root) it produces the same result:
>>
>> 1.
>> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn
>> install -U -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
>> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
>> [WARNING] The requested profile "ml" could not be activated because it
>> does not exist.
>>
>> 2.
>> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v
>> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
>> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
>> [WARNING] The requested profile "lgpl" could not be activated because it
>> does not exist.
>> [WARNING] The requested profile "examples" could not be activated because
>> it does not exist.
>> [WARNING] The requested profile "scala" could not be activated because it
>> does not exist.
>> [WARNING] The requested profile "ml" could not be activated because it
>> does not exist.

[Dmitriy Pavlov]

Hi Petr,

Thank you, it is great that you found the solution with low impact.

Lets create ticket and merge PR.

пт, 2 мар. 2018 г. в 10:06, Petr Ivanov <[hidden email]>:

> The problem is solved by updating flatten-maven-plugin version to 1.0.1.
>
> Nikolay, please, double check it.
> If it really solves the problem, please, fill the ticket (or point to
> existing one), so I can update it and check impact on release procedure.
>
>
>
> > On 1 Mar 2018, at 17:04, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Petr.
> >
> > Thank you for trying!
> >
> > Did you remove 'test' dependencies before running commands?
> >
> > Because I commit in master only correct pom.xml for current build
> process,
> > of course.
> >
> > But, to make things works I have to copy paste transitive dependencies
> from
> > spark-core.
> >
> > 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" <[hidden email]>
> > написал:
> >
> >>
> >>>
> >>> I don't get what is the point.
> >>> Did you try to reproduce issue?
> >>> Or should I provide full traces to you?
> >>>
> >>
> >> My point in inability to fully understand and describe the problem in
> >> terms of mechanism which causes it.
> >> For now I can see only indirect guessing.
> >>
> >>
> >> And yes, I’ve run your commands and both times (in spark module
> directory
> >> and in root) it produces the same result:
> >>
> >> 1.
> >> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn
> >> install -U -Plgpl,examples,scala,-clean-libs,-release,ml
> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
> >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> >> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> [WARNING] The requested profile "ml" could not be activated because it
> >> does not exist.
> >>
> >> 2.
> >> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v
> >> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml
> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
> >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> >> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> [WARNING] The requested profile "lgpl" could not be activated because it
> >> does not exist.
> >> [WARNING] The requested profile "examples" could not be activated
> because
> >> it does not exist.
> >> [WARNING] The requested profile "scala" could not be activated because
> it
> >> does not exist.
> >> [WARNING] The requested profile "ml" could not be activated because it
> >> does not exist.
>
>