Using IgniteContext in... well, Java context

[Konstantin Boudnik-2]

Guys,

I've tried to play a little bit with IgniteContext and the whole IgniteRDD
stuff from the perspective of not touching Scala ever again. And here's what I
have found: IgniteContext isn't usable from Java (or Groovy for that matter).
And it isn't an attempt to critique Ignite's RDD implementation, because we
have to follow the design patterns setup by the 3rd party platform, Spark in
this case.

If I want to submit a Java-based job into spark cluster I need to do something
like this:

import org.apache.ignite.spark.*
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext

SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
JavaSparkContext jsc = new JavaSparkContext(sparkConf)

so far so good. Now, if I want to use Ignite in it, I should be doing

def ic = new IgniteContext<T1, T2>(jsc, "spark-ignite-config.xml")

and that's where I hit the wall, because JavaSparkContext isn't a subtype of
SparkContext in Spark world, and I can not cast one to another.

Well then, perhaps I can use SparkContext and hope for the best? Turns out I
can not, because the use of the SparkContext gets me into the swamp of Scala
type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon the
day will come soon, when they will have Function27 and so on). This leads me
to believe the current implementation of IgniteRDD is only good to be used
from Scala, unless I am completely wrong and don't know what I am talking
about. Which might be quite possible, of course.

My question is very simple: is there a way to use IgniteRDD from Java-language
family, e.g. Java and Groovy, or there has to be JavaIgniteRDD implementation
of it much like the state of things is in the Spark itself?

Thanks for any feedback
--
  Cos

[Andrey Gura]

Cos,

Ignite provides Java friendly RDD API. See JavaIgniteContext and
JavaIgniteRDD classes.

On Fri, Oct 9, 2015 at 12:26 AM, Konstantin Boudnik <[hidden email]> wrote:

> Guys,
>
> I've tried to play a little bit with IgniteContext and the whole IgniteRDD
> stuff from the perspective of not touching Scala ever again. And here's
> what I
> have found: IgniteContext isn't usable from Java (or Groovy for that
> matter).
> And it isn't an attempt to critique Ignite's RDD implementation, because we
> have to follow the design patterns setup by the 3rd party platform, Spark
> in
> this case.
>
> If I want to submit a Java-based job into spark cluster I need to do
> something
> like this:
>
> import org.apache.ignite.spark.*
> import org.apache.spark.SparkConf
> import org.apache.spark.api.java.JavaSparkContext
>
> SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
> sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
> JavaSparkContext jsc = new JavaSparkContext(sparkConf)
>
> so far so good. Now, if I want to use Ignite in it, I should be doing
>
> def ic = new IgniteContext<T1, T2>(jsc, "spark-ignite-config.xml")
>
> and that's where I hit the wall, because JavaSparkContext isn't a subtype
> of
> SparkContext in Spark world, and I can not cast one to another.
>
> Well then, perhaps I can use SparkContext and hope for the best? Turns out
> I
> can not, because the use of the SparkContext gets me into the swamp of
> Scala
> type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon
> the
> day will come soon, when they will have Function27 and so on). This leads
> me
> to believe the current implementation of IgniteRDD is only good to be used
> from Scala, unless I am completely wrong and don't know what I am talking
> about. Which might be quite possible, of course.
>
> My question is very simple: is there a way to use IgniteRDD from
> Java-language
> family, e.g. Java and Groovy, or there has to be JavaIgniteRDD
> implementation
> of it much like the state of things is in the Spark itself?
>
> Thanks for any feedback
> --
>   Cos
>
>

--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

[Konstantin Boudnik-2]

Ah, missed that. Thanks!

It is pity to see how Scala based platform forces every downstream to treat
Java code like a second-class citizen. How nice ;(

Cos

On Fri, Oct 09, 2015 at 01:11PM, Andrey Gura wrote:

> Cos,
>
> Ignite provides Java friendly RDD API. See JavaIgniteContext and
> JavaIgniteRDD classes.
>
> On Fri, Oct 9, 2015 at 12:26 AM, Konstantin Boudnik <[hidden email]> wrote:
>
> > Guys,
> >
> > I've tried to play a little bit with IgniteContext and the whole IgniteRDD
> > stuff from the perspective of not touching Scala ever again. And here's
> > what I
> > have found: IgniteContext isn't usable from Java (or Groovy for that
> > matter).
> > And it isn't an attempt to critique Ignite's RDD implementation, because we
> > have to follow the design patterns setup by the 3rd party platform, Spark
> > in
> > this case.
> >
> > If I want to submit a Java-based job into spark cluster I need to do
> > something
> > like this:
> >
> > import org.apache.ignite.spark.*
> > import org.apache.spark.SparkConf
> > import org.apache.spark.api.java.JavaSparkContext
> >
> > SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
> > sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
> > JavaSparkContext jsc = new JavaSparkContext(sparkConf)
> >
> > so far so good. Now, if I want to use Ignite in it, I should be doing
> >
> > def ic = new IgniteContext<T1, T2>(jsc, "spark-ignite-config.xml")
> >
> > and that's where I hit the wall, because JavaSparkContext isn't a subtype
> > of
> > SparkContext in Spark world, and I can not cast one to another.
> >
> > Well then, perhaps I can use SparkContext and hope for the best? Turns out
> > I
> > can not, because the use of the SparkContext gets me into the swamp of
> > Scala
> > type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon
> > the
> > day will come soon, when they will have Function27 and so on). This leads
> > me
> > to believe the current implementation of IgniteRDD is only good to be used
> > from Scala, unless I am completely wrong and don't know what I am talking
> > about. Which might be quite possible, of course.
> >
> > My question is very simple: is there a way to use IgniteRDD from
> > Java-language
> > family, e.g. Java and Groovy, or there has to be JavaIgniteRDD
> > implementation
> > of it much like the state of things is in the Spark itself?
> >
> > Thanks for any feedback
> > --
> >   Cos
> >
> >
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com