|
|
Guys,
I've tried to play a little bit with IgniteContext and the whole IgniteRDD
stuff from the perspective of not touching Scala ever again. And here's what I
have found: IgniteContext isn't usable from Java (or Groovy for that matter).
And it isn't an attempt to critique Ignite's RDD implementation, because we
have to follow the design patterns setup by the 3rd party platform, Spark in
this case.
If I want to submit a Java-based job into spark cluster I need to do something
like this:
import org.apache.ignite.spark.*
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext
SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
JavaSparkContext jsc = new JavaSparkContext(sparkConf)
so far so good. Now, if I want to use Ignite in it, I should be doing
def ic = new IgniteContext<T1, T2>(jsc, "spark-ignite-config.xml")
and that's where I hit the wall, because JavaSparkContext isn't a subtype of
SparkContext in Spark world, and I can not cast one to another.
Well then, perhaps I can use SparkContext and hope for the best? Turns out I
can not, because the use of the SparkContext gets me into the swamp of Scala
type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon the
day will come soon, when they will have Function27 and so on). This leads me
to believe the current implementation of IgniteRDD is only good to be used
from Scala, unless I am completely wrong and don't know what I am talking
about. Which might be quite possible, of course.
My question is very simple: is there a way to use IgniteRDD from Java-language
family, e.g. Java and Groovy, or there has to be JavaIgniteRDD implementation
of it much like the state of things is in the Spark itself?
Thanks for any feedback
--
Cos
|
Locked
