Ignite-Spark integration meeting

[Alexey Zinoviev]

Hi, Igniters, and especially Nikolay Izhikov

I suggest to arrange the meeting via ASF-slack in #ignite-spark-integration
channel at 17 October, in 16 MSK Time.
The language of the meeting: Russian

*The topics to discuss:*

   - Review of next tickets


   1. [Spark][Bug] IgniteSpark integration forget to close the
      IgniteContext and stops the client node in case if error during
      PairFunction logic
      <https://issues.apache.org/jira/browse/IGNITE-11724>
      2. [Spark] Add initial support of Spark 2.4
      <https://issues.apache.org/jira/browse/IGNITE-12247>
      3. [Spark] Add joins to Spark Dataframe examples
      <https://issues.apache.org/jira/browse/IGNITE-9732>


   - Known issues of migration to 2.4 version
   - Organization of multiple version support for Apache Spark

P.S. meeting notes will be shared with the community
P.P.S Nikolay, if you have any objections about the time or topics, please
let me know

[Nikolay Izhikov-2]

Hello, Alexey.

I'm ready to take part in the meeting.
It will be in Russian, isn't it?

В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:

> Hi, Igniters, and especially Nikolay Izhikov
>
> I suggest to arrange the meeting via ASF-slack in #ignite-spark-integration
> channel at 17 October, in 16 MSK Time.
> The language of the meeting: Russian
>
> *The topics to discuss:*
>
>    - Review of next tickets
>
>
>    1. [Spark][Bug] IgniteSpark integration forget to close the
>       IgniteContext and stops the client node in case if error during
>       PairFunction logic
>       <https://issues.apache.org/jira/browse/IGNITE-11724>
>       2. [Spark] Add initial support of Spark 2.4
>       <https://issues.apache.org/jira/browse/IGNITE-12247>
>       3. [Spark] Add joins to Spark Dataframe examples
>       <https://issues.apache.org/jira/browse/IGNITE-9732>
>
>
>    - Known issues of migration to 2.4 version
>    - Organization of multiple version support for Apache Spark
>
> P.S. meeting notes will be shared with the community
> P.P.S Nikolay, if you have any objections about the time or topics, please
> let me know

[Alexey Zinoviev]

Yes, let's use the Russian for short discussion, the meeting notes will be
presented in English

пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]>:

> Hello, Alexey.
>
> I'm ready to take part in the meeting.
> It will be in Russian, isn't it?
>
> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > Hi, Igniters, and especially Nikolay Izhikov
> >
> > I suggest to arrange the meeting via ASF-slack in
> #ignite-spark-integration
> > channel at 17 October, in 16 MSK Time.
> > The language of the meeting: Russian
> >
> > *The topics to discuss:*
> >
> >    - Review of next tickets
> >
> >
> >    1. [Spark][Bug] IgniteSpark integration forget to close the
> >       IgniteContext and stops the client node in case if error during
> >       PairFunction logic
> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> >       2. [Spark] Add initial support of Spark 2.4
> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> >       3. [Spark] Add joins to Spark Dataframe examples
> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> >
> >
> >    - Known issues of migration to 2.4 version
> >    - Organization of multiple version support for Apache Spark
> >
> > P.S. meeting notes will be shared with the community
> > P.P.S Nikolay, if you have any objections about the time or topics,
> please
> > let me know
>

[Alexey Zinoviev]

The meeting notes and results of discussion are next:

   1. We are going to release Apache Ignite 2.8 with limited support of
   Spark 2.4.4 (known issues are listed here
   https://issues.apache.org/jira/browse/IGNITE-12054)
   2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
   fixed bugs
   3. We will create the developer-profiles spark_2_3 and spark_2_4
   separately from current scala profile
   4. We are ready to move the Ignite-Spark integration to the separate
   module after release 2.8 with support of most actual Spark versions and
   with independent release cycle
   5. We will work together on the next release roadmap with Nikolay and
   all interested igniters to improve the current solution (add new features
   as thin client support, working with caches via binary object and so on)
   6. Nikolay is ready to help with the review of tickets as an author of
   this component

Thanks



пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <[hidden email]>:

> Yes, let's use the Russian for short discussion, the meeting notes will be
> presented in English
>
> пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]>:
>
>> Hello, Alexey.
>>
>> I'm ready to take part in the meeting.
>> It will be in Russian, isn't it?
>>
>> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
>> > Hi, Igniters, and especially Nikolay Izhikov
>> >
>> > I suggest to arrange the meeting via ASF-slack in
>> #ignite-spark-integration
>> > channel at 17 October, in 16 MSK Time.
>> > The language of the meeting: Russian
>> >
>> > *The topics to discuss:*
>> >
>> >    - Review of next tickets
>> >
>> >
>> >    1. [Spark][Bug] IgniteSpark integration forget to close the
>> >       IgniteContext and stops the client node in case if error during
>> >       PairFunction logic
>> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
>> >       2. [Spark] Add initial support of Spark 2.4
>> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
>> >       3. [Spark] Add joins to Spark Dataframe examples
>> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
>> >
>> >
>> >    - Known issues of migration to 2.4 version
>> >    - Organization of multiple version support for Apache Spark
>> >
>> > P.S. meeting notes will be shared with the community
>> > P.P.S Nikolay, if you have any objections about the time or topics,
>> please
>> > let me know
>>
>

[dmagda]

>
>  1. We are going to release Apache Ignite 2.8 with limited support of
>    Spark 2.4.4 (known issues are listed here
>    https://issues.apache.org/jira/browse/IGNITE-12054)


What does limited support mean? Is it about regressions (something that
worked before but will fail) or capabilities unsupported even for the
presently released version of the integration?

-
Denis


On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <[hidden email]>
wrote:

> The meeting notes and results of discussion are next:
>
>    1. We are going to release Apache Ignite 2.8 with limited support of
>    Spark 2.4.4 (known issues are listed here
>    https://issues.apache.org/jira/browse/IGNITE-12054)
>    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
>    fixed bugs
>    3. We will create the developer-profiles spark_2_3 and spark_2_4
>    separately from current scala profile
>    4. We are ready to move the Ignite-Spark integration to the separate
>    module after release 2.8 with support of most actual Spark versions and
>    with independent release cycle
>    5. We will work together on the next release roadmap with Nikolay and
>    all interested igniters to improve the current solution (add new
> features
>    as thin client support, working with caches via binary object and so on)
>    6. Nikolay is ready to help with the review of tickets as an author of
>    this component
>
> Thanks
>
>
>
> пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <[hidden email]>:
>
> > Yes, let's use the Russian for short discussion, the meeting notes will
> be
> > presented in English
> >
> > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]>:
> >
> >> Hello, Alexey.
> >>
> >> I'm ready to take part in the meeting.
> >> It will be in Russian, isn't it?
> >>
> >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> >> > Hi, Igniters, and especially Nikolay Izhikov
> >> >
> >> > I suggest to arrange the meeting via ASF-slack in
> >> #ignite-spark-integration
> >> > channel at 17 October, in 16 MSK Time.
> >> > The language of the meeting: Russian
> >> >
> >> > *The topics to discuss:*
> >> >
> >> >    - Review of next tickets
> >> >
> >> >
> >> >    1. [Spark][Bug] IgniteSpark integration forget to close the
> >> >       IgniteContext and stops the client node in case if error during
> >> >       PairFunction logic
> >> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> >> >       2. [Spark] Add initial support of Spark 2.4
> >> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> >> >       3. [Spark] Add joins to Spark Dataframe examples
> >> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> >> >
> >> >
> >> >    - Known issues of migration to 2.4 version
> >> >    - Organization of multiple version support for Apache Spark
> >> >
> >> > P.S. meeting notes will be shared with the community
> >> > P.P.S Nikolay, if you have any objections about the time or topics,
> >> please
> >> > let me know
> >>
> >
>

[Ivan Pavlukhin]

Alexey,

Thank you for sharing the summary!

Small question. How are you going to integrations with different Spark versions?

чт, 17 окт. 2019 г. в 22:30, Denis Magda <[hidden email]>:

>
> >
> >  1. We are going to release Apache Ignite 2.8 with limited support of
> >    Spark 2.4.4 (known issues are listed here
> >    https://issues.apache.org/jira/browse/IGNITE-12054)
>
>
> What does limited support mean? Is it about regressions (something that
> worked before but will fail) or capabilities unsupported even for the
> presently released version of the integration?
>
> -
> Denis
>
>
> On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <[hidden email]>
> wrote:
>
> > The meeting notes and results of discussion are next:
> >
> >    1. We are going to release Apache Ignite 2.8 with limited support of
> >    Spark 2.4.4 (known issues are listed here
> >    https://issues.apache.org/jira/browse/IGNITE-12054)
> >    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
> >    fixed bugs
> >    3. We will create the developer-profiles spark_2_3 and spark_2_4
> >    separately from current scala profile
> >    4. We are ready to move the Ignite-Spark integration to the separate
> >    module after release 2.8 with support of most actual Spark versions and
> >    with independent release cycle
> >    5. We will work together on the next release roadmap with Nikolay and
> >    all interested igniters to improve the current solution (add new
> > features
> >    as thin client support, working with caches via binary object and so on)
> >    6. Nikolay is ready to help with the review of tickets as an author of
> >    this component
> >
> > Thanks
> >
> >
> >
> > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <[hidden email]>:
> >
> > > Yes, let's use the Russian for short discussion, the meeting notes will
> > be
> > > presented in English
> > >
> > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]>:
> > >
> > >> Hello, Alexey.
> > >>
> > >> I'm ready to take part in the meeting.
> > >> It will be in Russian, isn't it?
> > >>
> > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > >> > Hi, Igniters, and especially Nikolay Izhikov
> > >> >
> > >> > I suggest to arrange the meeting via ASF-slack in
> > >> #ignite-spark-integration
> > >> > channel at 17 October, in 16 MSK Time.
> > >> > The language of the meeting: Russian
> > >> >
> > >> > *The topics to discuss:*
> > >> >
> > >> >    - Review of next tickets
> > >> >
> > >> >
> > >> >    1. [Spark][Bug] IgniteSpark integration forget to close the
> > >> >       IgniteContext and stops the client node in case if error during
> > >> >       PairFunction logic
> > >> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > >> >       2. [Spark] Add initial support of Spark 2.4
> > >> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > >> >       3. [Spark] Add joins to Spark Dataframe examples
> > >> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > >> >
> > >> >
> > >> >    - Known issues of migration to 2.4 version
> > >> >    - Organization of multiple version support for Apache Spark
> > >> >
> > >> > P.S. meeting notes will be shared with the community
> > >> > P.P.S Nikolay, if you have any objections about the time or topics,
> > >> please
> > >> > let me know
> > >>
> > >
> >

--
Best regards,
Ivan Pavlukhin

[Ivan Pavlukhin]

Correction.

How are you going to TEST integrations with different Spark versions?

ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <[hidden email]>:

>
> Alexey,
>
> Thank you for sharing the summary!
>
> Small question. How are you going to integrations with different Spark versions?
>
> чт, 17 окт. 2019 г. в 22:30, Denis Magda <[hidden email]>:
> >
> > >
> > >  1. We are going to release Apache Ignite 2.8 with limited support of
> > >    Spark 2.4.4 (known issues are listed here
> > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> >
> >
> > What does limited support mean? Is it about regressions (something that
> > worked before but will fail) or capabilities unsupported even for the
> > presently released version of the integration?
> >
> > -
> > Denis
> >
> >
> > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <[hidden email]>
> > wrote:
> >
> > > The meeting notes and results of discussion are next:
> > >
> > >    1. We are going to release Apache Ignite 2.8 with limited support of
> > >    Spark 2.4.4 (known issues are listed here
> > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > >    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
> > >    fixed bugs
> > >    3. We will create the developer-profiles spark_2_3 and spark_2_4
> > >    separately from current scala profile
> > >    4. We are ready to move the Ignite-Spark integration to the separate
> > >    module after release 2.8 with support of most actual Spark versions and
> > >    with independent release cycle
> > >    5. We will work together on the next release roadmap with Nikolay and
> > >    all interested igniters to improve the current solution (add new
> > > features
> > >    as thin client support, working with caches via binary object and so on)
> > >    6. Nikolay is ready to help with the review of tickets as an author of
> > >    this component
> > >
> > > Thanks
> > >
> > >
> > >
> > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <[hidden email]>:
> > >
> > > > Yes, let's use the Russian for short discussion, the meeting notes will
> > > be
> > > > presented in English
> > > >
> > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]>:
> > > >
> > > >> Hello, Alexey.
> > > >>
> > > >> I'm ready to take part in the meeting.
> > > >> It will be in Russian, isn't it?
> > > >>
> > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > >> >
> > > >> > I suggest to arrange the meeting via ASF-slack in
> > > >> #ignite-spark-integration
> > > >> > channel at 17 October, in 16 MSK Time.
> > > >> > The language of the meeting: Russian
> > > >> >
> > > >> > *The topics to discuss:*
> > > >> >
> > > >> >    - Review of next tickets
> > > >> >
> > > >> >
> > > >> >    1. [Spark][Bug] IgniteSpark integration forget to close the
> > > >> >       IgniteContext and stops the client node in case if error during
> > > >> >       PairFunction logic
> > > >> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > >> >       2. [Spark] Add initial support of Spark 2.4
> > > >> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > >> >       3. [Spark] Add joins to Spark Dataframe examples
> > > >> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > >> >
> > > >> >
> > > >> >    - Known issues of migration to 2.4 version
> > > >> >    - Organization of multiple version support for Apache Spark
> > > >> >
> > > >> > P.S. meeting notes will be shared with the community
> > > >> > P.P.S Nikolay, if you have any objections about the time or topics,
> > > >> please
> > > >> > let me know
> > > >>
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin

--
Best regards,
Ivan Pavlukhin

[Alexey Zinoviev]

What kind of testing do you mean?  JUnit, integration via example or
something else?

ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <[hidden email]>:

> Correction.
>
> How are you going to TEST integrations with different Spark versions?
>
> ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <[hidden email]>:
> >
> > Alexey,
> >
> > Thank you for sharing the summary!
> >
> > Small question. How are you going to integrations with different Spark
> versions?
> >
> > чт, 17 окт. 2019 г. в 22:30, Denis Magda <[hidden email]>:
> > >
> > > >
> > > >  1. We are going to release Apache Ignite 2.8 with limited support of
> > > >    Spark 2.4.4 (known issues are listed here
> > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > >
> > >
> > > What does limited support mean? Is it about regressions (something that
> > > worked before but will fail) or capabilities unsupported even for the
> > > presently released version of the integration?
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> [hidden email]>
> > > wrote:
> > >
> > > > The meeting notes and results of discussion are next:
> > > >
> > > >    1. We are going to release Apache Ignite 2.8 with limited support
> of
> > > >    Spark 2.4.4 (known issues are listed here
> > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > >    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with
> known
> > > >    fixed bugs
> > > >    3. We will create the developer-profiles spark_2_3 and spark_2_4
> > > >    separately from current scala profile
> > > >    4. We are ready to move the Ignite-Spark integration to the
> separate
> > > >    module after release 2.8 with support of most actual Spark
> versions and
> > > >    with independent release cycle
> > > >    5. We will work together on the next release roadmap with Nikolay
> and
> > > >    all interested igniters to improve the current solution (add new
> > > > features
> > > >    as thin client support, working with caches via binary object and
> so on)
> > > >    6. Nikolay is ready to help with the review of tickets as an
> author of
> > > >    this component
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <[hidden email]
> >:
> > > >
> > > > > Yes, let's use the Russian for short discussion, the meeting notes
> will
> > > > be
> > > > > presented in English
> > > > >
> > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]
> >:
> > > > >
> > > > >> Hello, Alexey.
> > > > >>
> > > > >> I'm ready to take part in the meeting.
> > > > >> It will be in Russian, isn't it?
> > > > >>
> > > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > > >> >
> > > > >> > I suggest to arrange the meeting via ASF-slack in
> > > > >> #ignite-spark-integration
> > > > >> > channel at 17 October, in 16 MSK Time.
> > > > >> > The language of the meeting: Russian
> > > > >> >
> > > > >> > *The topics to discuss:*
> > > > >> >
> > > > >> >    - Review of next tickets
> > > > >> >
> > > > >> >
> > > > >> >    1. [Spark][Bug] IgniteSpark integration forget to close the
> > > > >> >       IgniteContext and stops the client node in case if error
> during
> > > > >> >       PairFunction logic
> > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > >> >       2. [Spark] Add initial support of Spark 2.4
> > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > >> >       3. [Spark] Add joins to Spark Dataframe examples
> > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > >> >
> > > > >> >
> > > > >> >    - Known issues of migration to 2.4 version
> > > > >> >    - Organization of multiple version support for Apache Spark
> > > > >> >
> > > > >> > P.S. meeting notes will be shared with the community
> > > > >> > P.P.S Nikolay, if you have any objections about the time or
> topics,
> > > > >> please
> > > > >> > let me know
> > > > >>
> > > > >
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[Ivan Pavlukhin]

Actually, I meant running tests on TeamCity.

ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <[hidden email]>:

>
> What kind of testing do you mean?  JUnit, integration via example or
> something else?
>
> ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <[hidden email]>:
>
> > Correction.
> >
> > How are you going to TEST integrations with different Spark versions?
> >
> > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <[hidden email]>:
> > >
> > > Alexey,
> > >
> > > Thank you for sharing the summary!
> > >
> > > Small question. How are you going to integrations with different Spark
> > versions?
> > >
> > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <[hidden email]>:
> > > >
> > > > >
> > > > >  1. We are going to release Apache Ignite 2.8 with limited support of
> > > > >    Spark 2.4.4 (known issues are listed here
> > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > >
> > > >
> > > > What does limited support mean? Is it about regressions (something that
> > > > worked before but will fail) or capabilities unsupported even for the
> > > > presently released version of the integration?
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > The meeting notes and results of discussion are next:
> > > > >
> > > > >    1. We are going to release Apache Ignite 2.8 with limited support
> > of
> > > > >    Spark 2.4.4 (known issues are listed here
> > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > >    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with
> > known
> > > > >    fixed bugs
> > > > >    3. We will create the developer-profiles spark_2_3 and spark_2_4
> > > > >    separately from current scala profile
> > > > >    4. We are ready to move the Ignite-Spark integration to the
> > separate
> > > > >    module after release 2.8 with support of most actual Spark
> > versions and
> > > > >    with independent release cycle
> > > > >    5. We will work together on the next release roadmap with Nikolay
> > and
> > > > >    all interested igniters to improve the current solution (add new
> > > > > features
> > > > >    as thin client support, working with caches via binary object and
> > so on)
> > > > >    6. Nikolay is ready to help with the review of tickets as an
> > author of
> > > > >    this component
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <[hidden email]
> > >:
> > > > >
> > > > > > Yes, let's use the Russian for short discussion, the meeting notes
> > will
> > > > > be
> > > > > > presented in English
> > > > > >
> > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <[hidden email]
> > >:
> > > > > >
> > > > > >> Hello, Alexey.
> > > > > >>
> > > > > >> I'm ready to take part in the meeting.
> > > > > >> It will be in Russian, isn't it?
> > > > > >>
> > > > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > > > >> >
> > > > > >> > I suggest to arrange the meeting via ASF-slack in
> > > > > >> #ignite-spark-integration
> > > > > >> > channel at 17 October, in 16 MSK Time.
> > > > > >> > The language of the meeting: Russian
> > > > > >> >
> > > > > >> > *The topics to discuss:*
> > > > > >> >
> > > > > >> >    - Review of next tickets
> > > > > >> >
> > > > > >> >
> > > > > >> >    1. [Spark][Bug] IgniteSpark integration forget to close the
> > > > > >> >       IgniteContext and stops the client node in case if error
> > during
> > > > > >> >       PairFunction logic
> > > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > >> >       2. [Spark] Add initial support of Spark 2.4
> > > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > >> >       3. [Spark] Add joins to Spark Dataframe examples
> > > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > >> >
> > > > > >> >
> > > > > >> >    - Known issues of migration to 2.4 version
> > > > > >> >    - Organization of multiple version support for Apache Spark
> > > > > >> >
> > > > > >> > P.S. meeting notes will be shared with the community
> > > > > >> > P.P.S Nikolay, if you have any objections about the time or
> > topics,
> > > > > >> please
> > > > > >> > let me know
> > > > > >>
> > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >

--
Best regards,
Ivan Pavlukhin

[Alexey Zinoviev]

Good point, now they are not in the Best state, we have a few ideas how to
fix, will discuss it later here in this thread.

If you have any objection write it, but my position - creation special
Spark Visa on TC, like for ML, maybe two for different Spark versions, and
all commits to Spark will require this Visa.


ср, 23 окт. 2019 г., 16:25 Ivan Pavlukhin <[hidden email]>:

> Actually, I meant running tests on TeamCity.
>
> ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <[hidden email]>:
> >
> > What kind of testing do you mean?  JUnit, integration via example or
> > something else?
> >
> > ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <[hidden email]>:
> >
> > > Correction.
> > >
> > > How are you going to TEST integrations with different Spark versions?
> > >
> > > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <[hidden email]>:
> > > >
> > > > Alexey,
> > > >
> > > > Thank you for sharing the summary!
> > > >
> > > > Small question. How are you going to integrations with different
> Spark
> > > versions?
> > > >
> > > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <[hidden email]>:
> > > > >
> > > > > >
> > > > > >  1. We are going to release Apache Ignite 2.8 with limited
> support of
> > > > > >    Spark 2.4.4 (known issues are listed here
> > > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > >
> > > > >
> > > > > What does limited support mean? Is it about regressions (something
> that
> > > > > worked before but will fail) or capabilities unsupported even for
> the
> > > > > presently released version of the integration?
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > The meeting notes and results of discussion are next:
> > > > > >
> > > > > >    1. We are going to release Apache Ignite 2.8 with limited
> support
> > > of
> > > > > >    Spark 2.4.4 (known issues are listed here
> > > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > >    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4
> with
> > > known
> > > > > >    fixed bugs
> > > > > >    3. We will create the developer-profiles spark_2_3 and
> spark_2_4
> > > > > >    separately from current scala profile
> > > > > >    4. We are ready to move the Ignite-Spark integration to the
> > > separate
> > > > > >    module after release 2.8 with support of most actual Spark
> > > versions and
> > > > > >    with independent release cycle
> > > > > >    5. We will work together on the next release roadmap with
> Nikolay
> > > and
> > > > > >    all interested igniters to improve the current solution (add
> new
> > > > > > features
> > > > > >    as thin client support, working with caches via binary object
> and
> > > so on)
> > > > > >    6. Nikolay is ready to help with the review of tickets as an
> > > author of
> > > > > >    this component
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > >
> > > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <
> [hidden email]
> > > >:
> > > > > >
> > > > > > > Yes, let's use the Russian for short discussion, the meeting
> notes
> > > will
> > > > > > be
> > > > > > > presented in English
> > > > > > >
> > > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <
> [hidden email]
> > > >:
> > > > > > >
> > > > > > >> Hello, Alexey.
> > > > > > >>
> > > > > > >> I'm ready to take part in the meeting.
> > > > > > >> It will be in Russian, isn't it?
> > > > > > >>
> > > > > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > > > > >> >
> > > > > > >> > I suggest to arrange the meeting via ASF-slack in
> > > > > > >> #ignite-spark-integration
> > > > > > >> > channel at 17 October, in 16 MSK Time.
> > > > > > >> > The language of the meeting: Russian
> > > > > > >> >
> > > > > > >> > *The topics to discuss:*
> > > > > > >> >
> > > > > > >> >    - Review of next tickets
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >    1. [Spark][Bug] IgniteSpark integration forget to close
> the
> > > > > > >> >       IgniteContext and stops the client node in case if
> error
> > > during
> > > > > > >> >       PairFunction logic
> > > > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > > >> >       2. [Spark] Add initial support of Spark 2.4
> > > > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > > >> >       3. [Spark] Add joins to Spark Dataframe examples
> > > > > > >> >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >    - Known issues of migration to 2.4 version
> > > > > > >> >    - Organization of multiple version support for Apache
> Spark
> > > > > > >> >
> > > > > > >> > P.S. meeting notes will be shared with the community
> > > > > > >> > P.P.S Nikolay, if you have any objections about the time or
> > > topics,
> > > > > > >> please
> > > > > > >> > let me know
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[Nikolay Izhikov-2]

The only issue with the Spark integration tests - they should be runned under JDK8.
That's all.

I wrote about it recently [1]

[1] http://apache-ignite-developers.2346864.n4.nabble.com/Spark-examples-on-TC-td43591.html

В Ср, 23/10/2019 в 17:46 +0200, Alexey Zinoviev пишет:

> Good point, now they are not in the Best state, we have a few ideas how to
> fix, will discuss it later here in this thread.
>
> If you have any objection write it, but my position - creation special
> Spark Visa on TC, like for ML, maybe two for different Spark versions, and
> all commits to Spark will require this Visa.
>
>
> ср, 23 окт. 2019 г., 16:25 Ivan Pavlukhin <[hidden email]>:
>
> > Actually, I meant running tests on TeamCity.
> >
> > ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <[hidden email]>:
> > >
> > > What kind of testing do you mean?  JUnit, integration via example or
> > > something else?
> > >
> > > ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <[hidden email]>:
> > >
> > > > Correction.
> > > >
> > > > How are you going to TEST integrations with different Spark versions?
> > > >
> > > > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <[hidden email]>:
> > > > >
> > > > > Alexey,
> > > > >
> > > > > Thank you for sharing the summary!
> > > > >
> > > > > Small question. How are you going to integrations with different
> >
> > Spark
> > > > versions?
> > > > >
> > > > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <[hidden email]>:
> > > > > >
> > > > > > >
> > > > > > >  1. We are going to release Apache Ignite 2.8 with limited
> >
> > support of
> > > > > > >    Spark 2.4.4 (known issues are listed here
> > > > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > >
> > > > > >
> > > > > > What does limited support mean? Is it about regressions (something
> >
> > that
> > > > > > worked before but will fail) or capabilities unsupported even for
> >
> > the
> > > > > > presently released version of the integration?
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > > >
> > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > The meeting notes and results of discussion are next:
> > > > > > >
> > > > > > >    1. We are going to release Apache Ignite 2.8 with limited
> >
> > support
> > > > of
> > > > > > >    Spark 2.4.4 (known issues are listed here
> > > > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > > >    2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4
> >
> > with
> > > > known
> > > > > > >    fixed bugs
> > > > > > >    3. We will create the developer-profiles spark_2_3 and
> >
> > spark_2_4
> > > > > > >    separately from current scala profile
> > > > > > >    4. We are ready to move the Ignite-Spark integration to the
> > > >
> > > > separate
> > > > > > >    module after release 2.8 with support of most actual Spark
> > > >
> > > > versions and
> > > > > > >    with independent release cycle
> > > > > > >    5. We will work together on the next release roadmap with
> >
> > Nikolay
> > > > and
> > > > > > >    all interested igniters to improve the current solution (add
> >
> > new
> > > > > > > features
> > > > > > >    as thin client support, working with caches via binary object
> >
> > and
> > > > so on)
> > > > > > >    6. Nikolay is ready to help with the review of tickets as an
> > > >
> > > > author of
> > > > > > >    this component
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <
> >
> > [hidden email]
> > > > > :
> > > > > > >
> > > > > > > > Yes, let's use the Russian for short discussion, the meeting
> >
> > notes
> > > > will
> > > > > > > be
> > > > > > > > presented in English
> > > > > > > >
> > > > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <
> >
> > [hidden email]
> > > > > :
> > > > > > > >
> > > > > > > > > Hello, Alexey.
> > > > > > > > >
> > > > > > > > > I'm ready to take part in the meeting.
> > > > > > > > > It will be in Russian, isn't it?
> > > > > > > > >
> > > > > > > > > В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > > > > > > > Hi, Igniters, and especially Nikolay Izhikov
> > > > > > > > > >
> > > > > > > > > > I suggest to arrange the meeting via ASF-slack in
> > > > > > > > >
> > > > > > > > > #ignite-spark-integration
> > > > > > > > > > channel at 17 October, in 16 MSK Time.
> > > > > > > > > > The language of the meeting: Russian
> > > > > > > > > >
> > > > > > > > > > *The topics to discuss:*
> > > > > > > > > >
> > > > > > > > > >    - Review of next tickets
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >    1. [Spark][Bug] IgniteSpark integration forget to close
> >
> > the
> > > > > > > > > >       IgniteContext and stops the client node in case if
> >
> > error
> > > > during
> > > > > > > > > >       PairFunction logic
> > > > > > > > > >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > > > > > >       2. [Spark] Add initial support of Spark 2.4
> > > > > > > > > >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > > > > > >       3. [Spark] Add joins to Spark Dataframe examples
> > > > > > > > > >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >    - Known issues of migration to 2.4 version
> > > > > > > > > >    - Organization of multiple version support for Apache
> >
> > Spark
> > > > > > > > > >
> > > > > > > > > > P.S. meeting notes will be shared with the community
> > > > > > > > > > P.P.S Nikolay, if you have any objections about the time or
> > > >
> > > > topics,
> > > > > > > > > please
> > > > > > > > > > let me know
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >