[jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0

[Anton Vinogradov (Jira)]

Denis Magda created IGNITE-6163:
-----------------------------------

             Summary: Upgrade to Spark 2.2.0
                 Key: IGNITE-6163
                 URL: https://issues.apache.org/jira/browse/IGNITE-6163
             Project: Ignite
          Issue Type: Improvement
            Reporter: Denis Magda
            Priority: Blocker
             Fix For: 2.2


Apache Ignite's Spark integration module has to be updated to Spark 2.2.0 to avoid issues like this:
https://github.com/dmagda/IgniteSparkIoT/issues/1

In short, Spark 2.1.0 and Spark 2.2.0 modules can't be mixed together, otherwise, the exceptions like this will pop up:

{code}
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.setStreamingJobProgressListener(Lorg/apache/spark/scheduler/SparkListener;)V
        at org.apache.spark.streaming.ui.StreamingTab.<init>(StreamingTab.scala:41)
        at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:192)
        at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
        at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:138)
        at org.apache.ignite.iot.SparkStreamerStartup.main(SparkStreamerStartup.java:71)
17/08/22 18:27:25 INFO SparkContext: Invoking stop() from shutdown hook
17/08/22 18:27:25 INFO SparkUI: Stopped Spark web UI at http://10.0.1.6:4040
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[dmagda]

Seems it’s time to upgrade ignite-spark module once again. Soon we should start getting more issues related to Spark 2.2.0 incompatibility.

—
Denis

> Begin forwarded message:
>
> From: "Denis Magda (JIRA)" <[hidden email]>
> Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
> Date: August 22, 2017 at 5:26:00 PM PDT
> To: [hidden email]
> Reply-To: [hidden email]
>
> https://issues.apache.org/jira/browse/IGNITE-6163 <https://issues.apache.org/jira/browse/IGNITE-6163>

[dsetrakyan]

+1

On Tue, Aug 22, 2017 at 5:30 PM, Denis Magda <[hidden email]> wrote:

> Seems it’s time to upgrade ignite-spark module once again. Soon we should
> start getting more issues related to Spark 2.2.0 incompatibility.
>
> —
> Denis
>
> > Begin forwarded message:
> >
> > From: "Denis Magda (JIRA)" <[hidden email]>
> > Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
> > Date: August 22, 2017 at 5:26:00 PM PDT
> > To: [hidden email]
> > Reply-To: [hidden email]
> >
> > https://issues.apache.org/jira/browse/IGNITE-6163 <
> https://issues.apache.org/jira/browse/IGNITE-6163>
>

[Valentin Kulichenko]

Denis,

I just found out that with version 2.2.0 Spark dropped Java 7 support:
https://spark.apache.org/docs/2.2.0/

I'm not sure it's a good idea to do an upgrade as long as we support Java
7. Probably we should find a way to support both 2.1.0 and 2.2.0 in
parallel. What do you think?

-Val

On Tue, Aug 22, 2017 at 5:30 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> +1
>
> On Tue, Aug 22, 2017 at 5:30 PM, Denis Magda <[hidden email]> wrote:
>
> > Seems it’s time to upgrade ignite-spark module once again. Soon we should
> > start getting more issues related to Spark 2.2.0 incompatibility.
> >
> > —
> > Denis
> >
> > > Begin forwarded message:
> > >
> > > From: "Denis Magda (JIRA)" <[hidden email]>
> > > Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
> > > Date: August 22, 2017 at 5:26:00 PM PDT
> > > To: [hidden email]
> > > Reply-To: [hidden email]
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-6163 <
> > https://issues.apache.org/jira/browse/IGNITE-6163>
> >
>

[dmagda]

Opps,

It’s time for Ignite to say goodbye to Java 7 too. Hope this happens in the beginning of the next year.

—
Denis

> On Nov 29, 2017, at 3:08 PM, Valentin Kulichenko <[hidden email]> wrote:
>
> Denis,
>
> I just found out that with version 2.2.0 Spark dropped Java 7 support:
> https://spark.apache.org/docs/2.2.0/
>
> I'm not sure it's a good idea to do an upgrade as long as we support Java
> 7. Probably we should find a way to support both 2.1.0 and 2.2.0 in
> parallel. What do you think?
>
> -Val
>
> On Tue, Aug 22, 2017 at 5:30 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
>> +1
>>
>> On Tue, Aug 22, 2017 at 5:30 PM, Denis Magda <[hidden email]> wrote:
>>
>>> Seems it’s time to upgrade ignite-spark module once again. Soon we should
>>> start getting more issues related to Spark 2.2.0 incompatibility.
>>>
>>> —
>>> Denis
>>>
>>>> Begin forwarded message:
>>>>
>>>> From: "Denis Magda (JIRA)" <[hidden email]>
>>>> Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
>>>> Date: August 22, 2017 at 5:26:00 PM PDT
>>>> To: [hidden email]
>>>> Reply-To: [hidden email]
>>>>
>>>> https://issues.apache.org/jira/browse/IGNITE-6163 <
>>> https://issues.apache.org/jira/browse/IGNITE-6163>
>>>
>>