Adding Ignite as a data source needed in any case, because it's possible
(for example) that one will join an Ignite DataFrame with something else.
So this would be the first step, and it's comparatively easy, I think
couple of weeks of work.
However, I agree that without a custom strategy the whole integration
doesn't worth the effort. The biggest part here is query generator, and it
seems to be a pretty big project (couple of months?).
-Val
On Tue, Jan 3, 2017 at 5:07 PM, Denis Magda <
[hidden email]> wrote:
> Great drill down analysis. Thanks, Val!
>
> I think, that this kind of integration between Spark and Ignite will be
> truly useful if we support a logical to physical plan conversion executing
> a query directly over the data cached in Ignite. Not sure that it worth our
> efforts to support Ignite just as one more Spark data source if,
> eventually, Spark still will prepare RDDs internally by copying data at
> query execution time.
>
> Do you have an understanding on the efforts to support the logical to
> physical plan conversion? Do we need to ask Sergi or someone else from the
> community to investigate this deeper?
>
> —
> Denis
>
> > On Jan 2, 2017, at 9:39 PM, Valentin Kulichenko <
>
[hidden email]> wrote:
> >
> > Folks,
> >
> > I made some investigation in this area and put my thoughts in the ticket:
> >
https://issues.apache.org/jira/browse/IGNITE-3084> >
> > Can someone take a look and provide feedback? Feel free to ask questions
> if
> > anything is unclear.
> >
> > -Val
> >
> > On Tue, Apr 26, 2016 at 5:29 PM, Dmitriy Setrakyan <
>
[hidden email]>
> > wrote:
> >
> >> Igniters,
> >>
> >> Can someone provide some insight on what would it take to support the
> >> DataFrame API directly in Spark applications? I am not an expert in this
> >> part of the product, so it is very difficult for me to judge.
> >>
> >> Thanks,
> >> D.
> >>
>
>
Locked
