Spark/YARN

[Franciscus, Naden]

Hey guys,

So it looks like using Spark/Ignite on YARN together simply doesn't work.

Many of us have Hadoop appliances where we aren't allowed to install anything on the nodes. So the only option is YARN which
barring a few bugs seems to work okay. But the IgniteContext within Spark doesn't allow you to read configuration files from YARN.

So since you allow users to pass in an IgniteConfiguration we have tried to manually set configuration on the POJOs:
https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala

But during any Spark distributed operation it will attempt to serialise this which is not possible since most of the classes contained
within IgniteConfiguration e.g. TcpDiscoverySpi are not serializable.

I am going to go through and see how many classes will need to be marked serializable (could be dozens) but a call will need to be made:

 1.  Mark everything within IgniteConfiguration as serializable.
 2.  Force ALL users of IgniteContext to either read config from HDFS or from a Local Filesystem. Both will go through Spring layer.

What's the best way to get a decision on this ?

Cheers,
Naden

[Franciscus, Naden]

Just as a follow up to this:

IgniteConfiguration has properties e.g. CountdownLatch which aren't serializable and HDFS Path is not serializable either so there's no clean way to
construct a IgniteContext without breaking backwards compatibility.

Maybe update this page:
https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd

To mention that you can't use an IgniteContext within any Spark map, filter etc. So code such as this e.g. Using BinaryObjects will not work.

val pairRdd = rdd.map(x => {
  val builder = igniteConext.ignite.binary.builder("DT1")
  builder.setField("id", x.toString)
  builder.setField("name", "test-" + x.toString)
   val binObj = builder.build
  binObj
}).zipWithIndex.map(r => (r._2, r._1))

Shame that Ignite doesn’t really work on Spark. Hopefully one day !


From: Naden Franciscus <[hidden email]<mailto:[hidden email]>>
Date: Monday, 2 May 2016 at 4:32 PM
To: "[hidden email]<mailto:[hidden email]>" <[hidden email]<mailto:[hidden email]>>
Subject: Spark/YARN


Hey guys,

So it looks like using Spark/Ignite on YARN together simply doesn't work.

Many of us have Hadoop appliances where we aren't allowed to install anything on the nodes. So the only option is YARN which
barring a few bugs seems to work okay. But the IgniteContext within Spark doesn't allow you to read configuration files from YARN.

So since you allow users to pass in an IgniteConfiguration we have tried to manually set configuration on the POJOs:
https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala

But during any Spark distributed operation it will attempt to serialise this which is not possible since most of the classes contained
within IgniteConfiguration e.g. TcpDiscoverySpi are not serializable.

I am going to go through and see how many classes will need to be marked serializable (could be dozens) but a call will need to be made:

 1.  Mark everything within IgniteConfiguration as serializable.
 2.  Force ALL users of IgniteContext to either read config from HDFS or from a Local Filesystem. Both will go through Spring layer.

What's the best way to get a decision on this ?

Cheers,
Naden

[Alexey Goncharuk]

Hi Naden,

 But the IgniteContext within Spark doesn't allow you to read configuration
> files from YARN.
>

I am a little bit confused. Ignite can be configured via basic Spring XML,
and you can definitely read those XML files off HDFS or any other source.
Is there any reason why XML does not work for you?

Now it is impossible to make IgniteConfiguration serializable because it
contains such components as SPIs, which are non-serializable by it's nature.

[Franciscus, Naden]

Hey Alexey,

Sorry I meant HDFS :)

It would be good for IgniteContext to have the option to read
configuration from HDFS.
Again many users of Hadoop aren't going to have the ability to modify data
nodes to install
Ignite. But I will raise that as a separate issue.

The big issue is that there is a lot of code like this in documentation
and elsewhere:
ignite.cache("myCache").put(personId, person);

Which we can't use in any RDD/DF map, filter etc because neither
Ignite/IgniteCache are serializable.


Cheers,
Naden





On 4/05/2016, 5:42 AM, "Alexey Goncharuk" <[hidden email]>
wrote:

>Hi Naden,
>
> But the IgniteContext within Spark doesn't allow you to read
>configuration
>> files from YARN.
>>
>
>I am a little bit confused. Ignite can be configured via basic Spring XML,
>and you can definitely read those XML files off HDFS or any other source.
>Is there any reason why XML does not work for you?
>
>Now it is impossible to make IgniteConfiguration serializable because it
>contains such components as SPIs, which are non-serializable by it's
>nature.

[vijayendra bhati]

Hi Naden,I am just wondering why you want to start ignite node in Yarn.You can stary the independent ignite node.What benefit you are getting by making part of yarn cluster.I am asking this because i too had the same queries but latet on i switched to starting independent ignite node on the same nodes where my node managers are running.I guess this too will give data locality etc.Also you can create RDD and save data directly to cache using RDD,you dont need to explicitly invoke ignite() or start().I can share the code if you want.
Regards,Vij

Sent from Yahoo Mail on Android
 
  On Wed, 4 May, 2016 at 5:19 am, Franciscus, Naden<[hidden email]> wrote:  
Hey Alexey,

Sorry I meant HDFS :)

It would be good for IgniteContext to have the option to read
configuration from HDFS.
Again many users of Hadoop aren't going to have the ability to modify data
nodes to install
Ignite. But I will raise that as a separate issue.

The big issue is that there is a lot of code like this in documentation
and elsewhere:
ignite.cache("myCache").put(personId, person);

Which we can't use in any RDD/DF map, filter etc because neither
Ignite/IgniteCache are serializable.


Cheers,
Naden





On 4/05/2016, 5:42 AM, "Alexey Goncharuk" <[hidden email]>
wrote:

>Hi Naden,
>
> But the IgniteContext within Spark doesn't allow you to read
>configuration
>> files from YARN.
>>
>
>I am a little bit confused. Ignite can be configured via basic Spring XML,
>and you can definitely read those XML files off HDFS or any other source.
>Is there any reason why XML does not work for you?
>
>Now it is impossible to make IgniteConfiguration serializable because it
>contains such components as SPIs, which are non-serializable by it's
>nature.