[TeamCity] DataFrame Examples

[Nikolay Izhikov-2]

Hello, Igniters.

I've created several examples for a Ignite DataFrame.
Currently, they are not tested on Team City.

As far as I can understand, they should be checked in "Ignite Scalar Examples [Scala]".
But it marked as obsolete [1]

Please, give me advice:

1. Should we test Ignite DataFrame examples on Team City?

2. Which plan should be used for that?

[1] https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters

[Valentin Kulichenko]

Nikolay,

Spark integration is not related to Scalar, the only thing they have in
common is Scala. I think we should have a separate configuration for
ignite-spark module. If anything Spark related is currently in Scalar
suite, it should be moved from there.

-Val

On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <[hidden email]>
wrote:

> Hello, Igniters.
>
> I've created several examples for a Ignite DataFrame.
> Currently, they are not tested on Team City.
>
> As far as I can understand, they should be checked in "Ignite Scalar
> Examples [Scala]".
> But it marked as obsolete [1]
>
> Please, give me advice:
>
> 1. Should we test Ignite DataFrame examples on Team City?
>
> 2. Which plan should be used for that?
>
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters

[Dmitriy Pavlov]

Nikolay, Val,

Could these examples be tested in standard Ignite Examples test suite? As
far as I know Ignite ML and Ignite IGFS are also tested there.

Sincerely,
Dmitriy Pavlov

вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko <
[hidden email]>:

> Nikolay,
>
> Spark integration is not related to Scalar, the only thing they have in
> common is Scala. I think we should have a separate configuration for
> ignite-spark module. If anything Spark related is currently in Scalar
> suite, it should be moved from there.
>
> -Val
>
> On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <[hidden email]>
> wrote:
>
> > Hello, Igniters.
> >
> > I've created several examples for a Ignite DataFrame.
> > Currently, they are not tested on Team City.
> >
> > As far as I can understand, they should be checked in "Ignite Scalar
> > Examples [Scala]".
> > But it marked as obsolete [1]
> >
> > Please, give me advice:
> >
> > 1. Should we test Ignite DataFrame examples on Team City?
> >
> > 2. Which plan should be used for that?
> >
> > [1]
> https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> > IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters
>

[Valentin Kulichenko]

If we can run Scala examples there, then yes.

-Val

On Tue, Feb 27, 2018 at 11:54 AM, Dmitry Pavlov <[hidden email]>
wrote:

> Nikolay, Val,
>
> Could these examples be tested in standard Ignite Examples test suite? As
> far as I know Ignite ML and Ignite IGFS are also tested there.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko <
> [hidden email]>:
>
> > Nikolay,
> >
> > Spark integration is not related to Scalar, the only thing they have in
> > common is Scala. I think we should have a separate configuration for
> > ignite-spark module. If anything Spark related is currently in Scalar
> > suite, it should be moved from there.
> >
> > -Val
> >
> > On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <[hidden email]>
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I've created several examples for a Ignite DataFrame.
> > > Currently, they are not tested on Team City.
> > >
> > > As far as I can understand, they should be checked in "Ignite Scalar
> > > Examples [Scala]".
> > > But it marked as obsolete [1]
> > >
> > > Please, give me advice:
> > >
> > > 1. Should we test Ignite DataFrame examples on Team City?
> > >
> > > 2. Which plan should be used for that?
> > >
> > > [1]
> > https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> > > IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters
> >
>

[Nikolay Izhikov-2]

I've tested it locally so yes, Spark examples can be tested in Ignite Examples.

We need to enable `scala` profile for test plan and add `IgniteExamplesSparkSelfTestSuite` to Team City properties.

Dmitry, please, give me advice.
Should I edit Ignite Examples settings or should I create copy of test plan to adjust settings?



В Вт, 27/02/2018 в 19:54 +0000, Dmitry Pavlov пишет:

> Nikolay, Val,
>
> Could these examples be tested in standard Ignite Examples test suite? As
> far as I know Ignite ML and Ignite IGFS are also tested there.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko <
> [hidden email]>:
>
> > Nikolay,
> >
> > Spark integration is not related to Scalar, the only thing they have in
> > common is Scala. I think we should have a separate configuration for
> > ignite-spark module. If anything Spark related is currently in Scalar
> > suite, it should be moved from there.
> >
> > -Val
> >
> > On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <[hidden email]>
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I've created several examples for a Ignite DataFrame.
> > > Currently, they are not tested on Team City.
> > >
> > > As far as I can understand, they should be checked in "Ignite Scalar
> > > Examples [Scala]".
> > > But it marked as obsolete [1]
> > >
> > > Please, give me advice:
> > >
> > > 1. Should we test Ignite DataFrame examples on Team City?
> > >
> > > 2. Which plan should be used for that?
> > >
> > > [1]
> >
> > https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> > > IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters

[Dmitriy Pavlov]

I prefer to use existing profile. We have a lot of suites, and it is hard
to manage and monitor.

вт, 27 февр. 2018 г. в 23:02, Nikolay Izhikov <[hidden email]>:

> I've tested it locally so yes, Spark examples can be tested in Ignite
> Examples.
>
> We need to enable `scala` profile for test plan and add
> `IgniteExamplesSparkSelfTestSuite` to Team City properties.
>
> Dmitry, please, give me advice.
> Should I edit Ignite Examples settings or should I create copy of test
> plan to adjust settings?
>
>
>
> В Вт, 27/02/2018 в 19:54 +0000, Dmitry Pavlov пишет:
> > Nikolay, Val,
> >
> > Could these examples be tested in standard Ignite Examples test suite? As
> > far as I know Ignite ML and Ignite IGFS are also tested there.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko <
> > [hidden email]>:
> >
> > > Nikolay,
> > >
> > > Spark integration is not related to Scalar, the only thing they have in
> > > common is Scala. I think we should have a separate configuration for
> > > ignite-spark module. If anything Spark related is currently in Scalar
> > > suite, it should be moved from there.
> > >
> > > -Val
> > >
> > > On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <[hidden email]>
> > > wrote:
> > >
> > > > Hello, Igniters.
> > > >
> > > > I've created several examples for a Ignite DataFrame.
> > > > Currently, they are not tested on Team City.
> > > >
> > > > As far as I can understand, they should be checked in "Ignite Scalar
> > > > Examples [Scala]".
> > > > But it marked as obsolete [1]
> > > >
> > > > Please, give me advice:
> > > >
> > > > 1. Should we test Ignite DataFrame examples on Team City?
> > > >
> > > > 2. Which plan should be used for that?
> > > >
> > > > [1]
> > >
> > > https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> > > > IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters

[Nikolay Izhikov-2]

Hello, guys.

I think we are ready to enable testing of Spark examples on Team City.
Can you help me with best choice to do it?

Currently, we have 'Ignite Examples' to run `IgniteExamplesSelfTestSuite`.

We also have a `IgniteExamplesSparkSelfTestSuite` to test spark example.
`IgniteExamplesSparkSelfTestSuite` exists in source only if `spark` profile enabled so we can't just add it to `IgniteExamplesSelfTestSuite`.

I see following option here:

1. Edit 'Ignite Examples' to run both `IgniteExamplesSelfTestSuite` and `IgniteExamplesSparkSelfTestSuite`.
2. Create new test plan to run `IgniteExamplesSparkSelfTestSuite`.

Thoughts?

P.S. We also need minor changes [1] to make sure Spark examples are running correctly.

[1] https://github.com/apache/ignite/pull/3590

В Вт, 27/02/2018 в 11:56 -0800, Valentin Kulichenko пишет:

> If we can run Scala examples there, then yes.
>
> -Val
>
> On Tue, Feb 27, 2018 at 11:54 AM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Nikolay, Val,
> >
> > Could these examples be tested in standard Ignite Examples test suite? As
> > far as I know Ignite ML and Ignite IGFS are also tested there.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko <
> > [hidden email]>:
> >
> > > Nikolay,
> > >
> > > Spark integration is not related to Scalar, the only thing they have in
> > > common is Scala. I think we should have a separate configuration for
> > > ignite-spark module. If anything Spark related is currently in Scalar
> > > suite, it should be moved from there.
> > >
> > > -Val
> > >
> > > On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <[hidden email]>
> > > wrote:
> > >
> > > > Hello, Igniters.
> > > >
> > > > I've created several examples for a Ignite DataFrame.
> > > > Currently, they are not tested on Team City.
> > > >
> > > > As far as I can understand, they should be checked in "Ignite Scalar
> > > > Examples [Scala]".
> > > > But it marked as obsolete [1]
> > > >
> > > > Please, give me advice:
> > > >
> > > > 1. Should we test Ignite DataFrame examples on Team City?
> > > >
> > > > 2. Which plan should be used for that?
> > > >
> > > > [1]
> > >
> > > https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> > > > IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters

[Dmitriy Pavlov]

Hi Igniters,

 It seems to me it is better to keep one suite. It is reasonable to
separate suites if tests requires too much time to run.

Using the same suite will save remarkable amount of agent time.

Sincerely,
Dmitriy Pavlov

вс, 18 мар. 2018 г. в 19:34, Nikolay Izhikov <[hidden email]>:

> Hello, guys.
>
> I think we are ready to enable testing of Spark examples on Team City.
> Can you help me with best choice to do it?
>
> Currently, we have 'Ignite Examples' to run `IgniteExamplesSelfTestSuite`.
>
> We also have a `IgniteExamplesSparkSelfTestSuite` to test spark example.
> `IgniteExamplesSparkSelfTestSuite` exists in source only if `spark`
> profile enabled so we can't just add it to `IgniteExamplesSelfTestSuite`.
>
> I see following option here:
>
> 1. Edit 'Ignite Examples' to run both `IgniteExamplesSelfTestSuite` and
> `IgniteExamplesSparkSelfTestSuite`.
> 2. Create new test plan to run `IgniteExamplesSparkSelfTestSuite`.
>
> Thoughts?
>
> P.S. We also need minor changes [1] to make sure Spark examples are
> running correctly.
>
> [1] https://github.com/apache/ignite/pull/3590
>
> В Вт, 27/02/2018 в 11:56 -0800, Valentin Kulichenko пишет:
> > If we can run Scala examples there, then yes.
> >
> > -Val
> >
> > On Tue, Feb 27, 2018 at 11:54 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Nikolay, Val,
> > >
> > > Could these examples be tested in standard Ignite Examples test suite?
> As
> > > far as I know Ignite ML and Ignite IGFS are also tested there.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko <
> > > [hidden email]>:
> > >
> > > > Nikolay,
> > > >
> > > > Spark integration is not related to Scalar, the only thing they have
> in
> > > > common is Scala. I think we should have a separate configuration for
> > > > ignite-spark module. If anything Spark related is currently in Scalar
> > > > suite, it should be moved from there.
> > > >
> > > > -Val
> > > >
> > > > On Tue, Feb 27, 2018 at 3:03 AM, Nikolay Izhikov <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > I've created several examples for a Ignite DataFrame.
> > > > > Currently, they are not tested on Team City.
> > > > >
> > > > > As far as I can understand, they should be checked in "Ignite
> Scalar
> > > > > Examples [Scala]".
> > > > > But it marked as obsolete [1]
> > > > >
> > > > > Please, give me advice:
> > > > >
> > > > > 1. Should we test Ignite DataFrame examples on Team City?
> > > > >
> > > > > 2. Which plan should be used for that?
> > > > >
> > > > > [1]
> > > >
> > > >
> https://ci.ignite.apache.org/viewLog.html?buildId=1068099&buildTypeId=
> > > > > IgniteTestsForCodeCoverage_IgniteScalarExamples&tab=buildParameters

[vveider]

+1 to sticking to one suite for now.

Maybe someday when examples' tests will surpass adequate build time, we'll
consider splitting them.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Nikolay Izhikov-2]

Hello, guys.

I merged to master my fix of tests. [1]

I edited "Ignite Examples" configuration to Team City to enable testing of Spark examples.

1. `spark` profile enabled for test plan.
2. `IgniteExamplesSparkSelfTestSuite` added to tests suites list.

Test results [2] are OK.
Please, Team City gurus, validate my edits.

[1] https://github.com/apache/ignite/commit/ce6a15782c93a95c58ccb46c44f391b39d4c4ac9
[2] https://ci.ignite.apache.org/viewLog.html?buildId=1145334&buildTypeId=IgniteTests24Java8_IgniteExamples&tab=testsInfo

В Пн, 19/03/2018 в 06:12 -0700, vveider пишет:
> +1 to sticking to one suite for now.
>
> Maybe someday when examples' tests will surpass adequate build time, we'll
> consider splitting them.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[vveider]

Have you detached template for build Ignite Examples?

> On 19 Mar 2018, at 18:08, Nikolay Izhikov <[hidden email]> wrote:
>
> Hello, guys.
>
> I merged to master my fix of tests. [1]
>
> I edited "Ignite Examples" configuration to Team City to enable testing of Spark examples.
>
> 1. `spark` profile enabled for test plan.
> 2. `IgniteExamplesSparkSelfTestSuite` added to tests suites list.
>
> Test results [2] are OK.
> Please, Team City gurus, validate my edits.
>
> [1] https://github.com/apache/ignite/commit/ce6a15782c93a95c58ccb46c44f391b39d4c4ac9
> [2] https://ci.ignite.apache.org/viewLog.html?buildId=1145334&buildTypeId=IgniteTests24Java8_IgniteExamples&tab=testsInfo
>
> В Пн, 19/03/2018 в 06:12 -0700, vveider пишет:
>> +1 to sticking to one suite for now.
>>
>> Maybe someday when examples' tests will surpass adequate build time, we'll
>> consider splitting them.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Nikolay Izhikov-2]

Yes.

В Пн, 19/03/2018 в 18:13 +0300, Petr Ivanov пишет:

> Have you detached template for build Ignite Examples?
>
>
> > On 19 Mar 2018, at 18:08, Nikolay Izhikov <[hidden email]> wrote:
> >
> > Hello, guys.
> >
> > I merged to master my fix of tests. [1]
> >
> > I edited "Ignite Examples" configuration to Team City to enable testing of Spark examples.
> >
> > 1. `spark` profile enabled for test plan.
> > 2. `IgniteExamplesSparkSelfTestSuite` added to tests suites list.
> >
> > Test results [2] are OK.
> > Please, Team City gurus, validate my edits.
> >
> > [1] https://github.com/apache/ignite/commit/ce6a15782c93a95c58ccb46c44f391b39d4c4ac9
> > [2] https://ci.ignite.apache.org/viewLog.html?buildId=1145334&buildTypeId=IgniteTests24Java8_IgniteExamples&tab=testsInfo
> >
> > В Пн, 19/03/2018 в 06:12 -0700, vveider пишет:
> > > +1 to sticking to one suite for now.
> > >
> > > Maybe someday when examples' tests will surpass adequate build time, we'll
> > > consider splitting them.
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
>

[Nikolay Izhikov-2]

Petr, thanks for help with Team City configuration issue!

В Пн, 19/03/2018 в 18:14 +0300, Nikolay Izhikov пишет:

> Yes.
>
> В Пн, 19/03/2018 в 18:13 +0300, Petr Ivanov пишет:
> > Have you detached template for build Ignite Examples?
> >
> >
> > > On 19 Mar 2018, at 18:08, Nikolay Izhikov <[hidden email]> wrote:
> > >
> > > Hello, guys.
> > >
> > > I merged to master my fix of tests. [1]
> > >
> > > I edited "Ignite Examples" configuration to Team City to enable testing of Spark examples.
> > >
> > > 1. `spark` profile enabled for test plan.
> > > 2. `IgniteExamplesSparkSelfTestSuite` added to tests suites list.
> > >
> > > Test results [2] are OK.
> > > Please, Team City gurus, validate my edits.
> > >
> > > [1] https://github.com/apache/ignite/commit/ce6a15782c93a95c58ccb46c44f391b39d4c4ac9
> > > [2] https://ci.ignite.apache.org/viewLog.html?buildId=1145334&buildTypeId=IgniteTests24Java8_IgniteExamples&tab=testsInfo
> > >
> > > В Пн, 19/03/2018 в 06:12 -0700, vveider пишет:
> > > > +1 to sticking to one suite for now.
> > > >
> > > > Maybe someday when examples' tests will surpass adequate build time, we'll
> > > > consider splitting them.
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/