IgniteSparkSession Should Copy State on cloneSession()

[Stuart Macdonald]

Hello Igniters,

The IgniteSparkSession class extends SparkSession and overrides the
cloneSession() method. The contract for cloneSession() explicitly states
that it should clone all state (ie. the sharedState and sessionState
fields), however the IgniteSparkSession implementation doesn't clone its
state fields.


This has the side-effect of breaking stateful operations for anything which
uses cloneSession(), for example a Spark streaming job will not be able to
use cached data across streaming microbatches, which is a significant issue
for such applications.


I’ve created the following Jira for this issue:


https://issues.apache.org/jira/browse/IGNITE-9180


If there are no comments, I’ll create a pull request with a fix over the
next few days.


Stuart.

[Nikolay Izhikov-2]

Hello, Stuart.

Looks like a bug.

Please, go ahead with fixing it.

В Пт, 03/08/2018 в 12:43 +0100, Stuart Macdonald пишет:

> Hello Igniters,
>
> The IgniteSparkSession class extends SparkSession and overrides the
> cloneSession() method. The contract for cloneSession() explicitly states
> that it should clone all state (ie. the sharedState and sessionState
> fields), however the IgniteSparkSession implementation doesn't clone its
> state fields.
>
>
> This has the side-effect of breaking stateful operations for anything which
> uses cloneSession(), for example a Spark streaming job will not be able to
> use cached data across streaming microbatches, which is a significant issue
> for such applications.
>
>
> I’ve created the following Jira for this issue:
>
>
> https://issues.apache.org/jira/browse/IGNITE-9180
>
>
> If there are no comments, I’ll create a pull request with a fix over the
> next few days.
>
>
> Stuart.