Spark examples on TC

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark examples on TC

Nikolay Izhikov-2
Hello, Igniters.

Examples suite on TC doesn't work correctly under java9+. [1], [2]

The reason is the Spark examples.

Spark 2.3(our current version) work with scala 2.11 and scala 2.11 support of java9+ is "incomplete".

Can we stick example suite to jdk8 only?

[1] https://ci.ignite.apache.org/viewLog.html?buildId=4610507&buildTypeId=IgniteTests24Java8_Examples
[2] https://ci.ignite.apache.org/viewLog.html?buildId=4609013&buildTypeId=IgniteTests24Java8_Examples

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Spark examples on TC

Alexey Zinoviev
I support this solution, I think there is no issue to run with java 8 next
1-2 years all examples for TC purposes

чт, 19 сент. 2019 г. в 19:23, Nikolay Izhikov <[hidden email]>:

> Hello, Igniters.
>
> Examples suite on TC doesn't work correctly under java9+. [1], [2]
>
> The reason is the Spark examples.
>
> Spark 2.3(our current version) work with scala 2.11 and scala 2.11 support
> of java9+ is "incomplete".
>
> Can we stick example suite to jdk8 only?
>
> [1]
> https://ci.ignite.apache.org/viewLog.html?buildId=4610507&buildTypeId=IgniteTests24Java8_Examples
> [2]
> https://ci.ignite.apache.org/viewLog.html?buildId=4609013&buildTypeId=IgniteTests24Java8_Examples
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark examples on TC

dmagda
In reply to this post by Nikolay Izhikov-2
Nikolay,

What needs to be done to ensure Java 9++ works with Spark Examples? Do we
need to create another Spark module or drop some scala version?

-
Denis


On Thu, Sep 19, 2019 at 9:23 AM Nikolay Izhikov <[hidden email]> wrote:

> Hello, Igniters.
>
> Examples suite on TC doesn't work correctly under java9+. [1], [2]
>
> The reason is the Spark examples.
>
> Spark 2.3(our current version) work with scala 2.11 and scala 2.11 support
> of java9+ is "incomplete".
>
> Can we stick example suite to jdk8 only?
>
> [1]
> https://ci.ignite.apache.org/viewLog.html?buildId=4610507&buildTypeId=IgniteTests24Java8_Examples
> [2]
> https://ci.ignite.apache.org/viewLog.html?buildId=4609013&buildTypeId=IgniteTests24Java8_Examples
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark examples on TC

Nikolay Izhikov-2
Hello, Denis.

Spark uses Scala 2.11 [1]
Scala 2.11 compatible with the jdk version described on [2].

Few notes from the link:

> We recommend using Java 8 for compiling Scala code
> JDK 11 compatibility notes
> As of Scala 2.13.0, 2.12.8 and 2.11.12, JDK 11 support is incomplete

I also got several issues with the Spark itself when trying to run it on
the Java 11.
Please, take a look at [3], [4]

All in all, with the spark 2.3 we should stay on java8 to get things worked.

[1] http://spark.apache.org/docs/2.4.0/
[2] https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html
[3]
https://stackoverflow.com/questions/49961991/java-lang-illegalargumentexception-at-org-apache-xbean-asm5-classreader-initu?noredirect=1&lq=1
[4]
https://stackoverflow.com/questions/51352591/spark-java-illegalargumentexception-at-org-apache-xbean-asm5-classreader


чт, 19 сент. 2019 г. в 21:39, Denis Magda <[hidden email]>:

> Nikolay,
>
> What needs to be done to ensure Java 9++ works with Spark Examples? Do we
> need to create another Spark module or drop some scala version?
>
> -
> Denis
>
>
> On Thu, Sep 19, 2019 at 9:23 AM Nikolay Izhikov <[hidden email]>
> wrote:
>
> > Hello, Igniters.
> >
> > Examples suite on TC doesn't work correctly under java9+. [1], [2]
> >
> > The reason is the Spark examples.
> >
> > Spark 2.3(our current version) work with scala 2.11 and scala 2.11
> support
> > of java9+ is "incomplete".
> >
> > Can we stick example suite to jdk8 only?
> >
> > [1]
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=4610507&buildTypeId=IgniteTests24Java8_Examples
> > [2]
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=4609013&buildTypeId=IgniteTests24Java8_Examples
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark examples on TC

dmagda
Nikolay,

Thanks for the details. How about enforcing JDK 8 for the Spark integration
in general and run its tests on the VM only? We'll do this until Spark
community fully support JDK 11+.

As for Spark 2.4.0 module [1], how about releasing Ignite 2.7.7 with Spark
related changes only? As long as Ignite modularization is a long process,
we can follow your advice and create two more modules ignite_spark_24 and
ignite_spark_25 . Plus Alexey is working on extra Spark improvements.

[1] https://issues.apache.org/jira/browse/IGNITE-12054
-
Denis


On Thu, Sep 19, 2019 at 11:58 PM Nikolay Izhikov <[hidden email]>
wrote:

> Hello, Denis.
>
> Spark uses Scala 2.11 [1]
> Scala 2.11 compatible with the jdk version described on [2].
>
> Few notes from the link:
>
> > We recommend using Java 8 for compiling Scala code
> > JDK 11 compatibility notes
> > As of Scala 2.13.0, 2.12.8 and 2.11.12, JDK 11 support is incomplete
>
> I also got several issues with the Spark itself when trying to run it on
> the Java 11.
> Please, take a look at [3], [4]
>
> All in all, with the spark 2.3 we should stay on java8 to get things
> worked.
>
> [1] http://spark.apache.org/docs/2.4.0/
> [2] https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html
> [3]
>
> https://stackoverflow.com/questions/49961991/java-lang-illegalargumentexception-at-org-apache-xbean-asm5-classreader-initu?noredirect=1&lq=1
> [4]
>
> https://stackoverflow.com/questions/51352591/spark-java-illegalargumentexception-at-org-apache-xbean-asm5-classreader
>
>
> чт, 19 сент. 2019 г. в 21:39, Denis Magda <[hidden email]>:
>
> > Nikolay,
> >
> > What needs to be done to ensure Java 9++ works with Spark Examples? Do we
> > need to create another Spark module or drop some scala version?
> >
> > -
> > Denis
> >
> >
> > On Thu, Sep 19, 2019 at 9:23 AM Nikolay Izhikov <[hidden email]>
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > Examples suite on TC doesn't work correctly under java9+. [1], [2]
> > >
> > > The reason is the Spark examples.
> > >
> > > Spark 2.3(our current version) work with scala 2.11 and scala 2.11
> > support
> > > of java9+ is "incomplete".
> > >
> > > Can we stick example suite to jdk8 only?
> > >
> > > [1]
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=4610507&buildTypeId=IgniteTests24Java8_Examples
> > > [2]
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=4609013&buildTypeId=IgniteTests24Java8_Examples
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark examples on TC

vveider
I can help with JDK8 enforcement — just give me the links to target suites when agreed.


> On 25 Sep 2019, at 01:53, Denis Magda <[hidden email]> wrote:
>
> Nikolay,
>
> Thanks for the details. How about enforcing JDK 8 for the Spark integration
> in general and run its tests on the VM only? We'll do this until Spark
> community fully support JDK 11+.
>
> As for Spark 2.4.0 module [1], how about releasing Ignite 2.7.7 with Spark
> related changes only? As long as Ignite modularization is a long process,
> we can follow your advice and create two more modules ignite_spark_24 and
> ignite_spark_25 . Plus Alexey is working on extra Spark improvements.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12054
> -
> Denis
>
>
> On Thu, Sep 19, 2019 at 11:58 PM Nikolay Izhikov <[hidden email]>
> wrote:
>
>> Hello, Denis.
>>
>> Spark uses Scala 2.11 [1]
>> Scala 2.11 compatible with the jdk version described on [2].
>>
>> Few notes from the link:
>>
>>> We recommend using Java 8 for compiling Scala code
>>> JDK 11 compatibility notes
>>> As of Scala 2.13.0, 2.12.8 and 2.11.12, JDK 11 support is incomplete
>>
>> I also got several issues with the Spark itself when trying to run it on
>> the Java 11.
>> Please, take a look at [3], [4]
>>
>> All in all, with the spark 2.3 we should stay on java8 to get things
>> worked.
>>
>> [1] http://spark.apache.org/docs/2.4.0/
>> [2] https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html
>> [3]
>>
>> https://stackoverflow.com/questions/49961991/java-lang-illegalargumentexception-at-org-apache-xbean-asm5-classreader-initu?noredirect=1&lq=1
>> [4]
>>
>> https://stackoverflow.com/questions/51352591/spark-java-illegalargumentexception-at-org-apache-xbean-asm5-classreader
>>
>>
>> чт, 19 сент. 2019 г. в 21:39, Denis Magda <[hidden email]>:
>>
>>> Nikolay,
>>>
>>> What needs to be done to ensure Java 9++ works with Spark Examples? Do we
>>> need to create another Spark module or drop some scala version?
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Thu, Sep 19, 2019 at 9:23 AM Nikolay Izhikov <[hidden email]>
>>> wrote:
>>>
>>>> Hello, Igniters.
>>>>
>>>> Examples suite on TC doesn't work correctly under java9+. [1], [2]
>>>>
>>>> The reason is the Spark examples.
>>>>
>>>> Spark 2.3(our current version) work with scala 2.11 and scala 2.11
>>> support
>>>> of java9+ is "incomplete".
>>>>
>>>> Can we stick example suite to jdk8 only?
>>>>
>>>> [1]
>>>>
>>>
>> https://ci.ignite.apache.org/viewLog.html?buildId=4610507&buildTypeId=IgniteTests24Java8_Examples
>>>> [2]
>>>>
>>>
>> https://ci.ignite.apache.org/viewLog.html?buildId=4609013&buildTypeId=IgniteTests24Java8_Examples
>>>>
>>>
>>