My detailed issues as below this link, please refer to this link
http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tt6910.htmlMy question:
1) What ports should be opened on driver machine or yarn machine?(There issues were happned on our production env)
2) But in my local env(3 spark cluster[1 master, 2 workers] and 1 yarn cluster) and there is no issues as i manually closed my firewall, then fixed this issue.
I faced another issues as below
========================
1)
val igniteContext = new IgniteContext[Integer,Integer](sc, () => new IgniteConfiguration())
val sharedRDD = igniteContext.fromCache("sharedBaselineCacheRDD")
val initalRDD = sc.parallelize(1 to 100000,10).map(i => (new Integer(i), new Integer(i)))
println("initalRDD.couner=/. " + initalRDD.count() +"\tpartition=> " + initalRDD.partitions.size)
sharedRDD.savePairs(initalRDD, true)
println("=====>totalcounter" + sharedRDD.count + "\t paris => " + sharedRDD.partitions.size)
println("=====>" + sharedRDD.filter(_._2 > 50000).count)
2) submit a jar to yarn cluster as below command
/u01/spark-1.6.0-hive/bin/spark-submit --driver-memory 4G --class com.ValidSparkCache --master yarn --executor-cores 5 --executor-memory 1000m --num-executors 10 --conf spark.rdd.compress=false --conf spark.shuffle.compress=false --conf spark.broadcast.compress=false /u01/xxx/sparkignite.jar -quesize 10 -batchSize 5000 -writethread 30 -runningSeconds 20
3) final result should be 50000, but actullay result is 40000,3000(ramdom)
final result is 40000 NOT 50000, this is happened also on embed integration with spark.
Locked
