Apache Ignite as a distributed processing back-ends
Locked
 
|
Hello Apache Beam fellows!
We at Apache Ignite community came across your project and would be happy to integrate with it. In short, Ignite is a distributed database and computational platform that has its own map-reduce like component: https://apacheignite.readme.io/docs/compute-grid <https://apacheignite.readme.io/docs/compute-grid> The integration will give Beam users an ability to use Ignite as a distributed processing back-end system and database. How should we proceed? Please share any relevant information. — Denis Ignite PMC |
Re: Apache Ignite as a distributed processing back-ends
|
|
Hi
This sounds awesome to have an Ignite runner which could compete with hazelcast-jet. The entry point would be https://beam.apache.org/contribute/runner-guide/ IMHO. Being on Ignite cluster also opens a lot of doors - reusing the filesystem or distributed structures. Very exiting. Le 8 déc. 2017 05:46, "Denis Magda" <[hidden email]> a écrit : > Hello Apache Beam fellows! > > We at Apache Ignite community came across your project and would be happy > to integrate with it. > > In short, Ignite is a distributed database and computational platform that > has its own map-reduce like component: > https://apacheignite.readme.io/docs/compute-grid > > The integration will give Beam users an ability to use Ignite as a > distributed processing back-end system and database. > > How should we proceed? Please share any relevant information. > > — > Denis > Ignite PMC > |
|
|
Hi Romain,
Thanks for the reference. Do you prefer to have the Ignite runner in Beam’s code base? From what I see, the current runners are hosted there: https://github.com/apache/beam/tree/master/runners <https://github.com/apache/beam/tree/master/runners> As for Ignite community, we would prefer to hold the integration in your repo. — Denis > On Dec 7, 2017, at 9:54 PM, Romain Manni-Bucau <[hidden email]> wrote: > > Hi > > This sounds awesome to have an Ignite runner which could compete with > hazelcast-jet. > > The entry point would be https://beam.apache.org/contribute/runner-guide/ > IMHO. > > Being on Ignite cluster also opens a lot of doors - reusing the filesystem > or distributed structures. Very exiting. > > Le 8 déc. 2017 05:46, "Denis Magda" <[hidden email]> a écrit : > >> Hello Apache Beam fellows! >> >> We at Apache Ignite community came across your project and would be happy >> to integrate with it. >> >> In short, Ignite is a distributed database and computational platform that >> has its own map-reduce like component: >> https://apacheignite.readme.io/docs/compute-grid >> >> The integration will give Beam users an ability to use Ignite as a >> distributed processing back-end system and database. >> >> How should we proceed? Please share any relevant information. >> >> — >> Denis >> Ignite PMC >> |
Re: Apache Ignite as a distributed processing back-ends
|
|
Having it inside the Apache Beam repo makes sense and I could see it being
a good fit as an IO and as a runner. On Tue, Dec 12, 2017 at 3:29 PM, Denis Magda <[hidden email]> wrote: > Hi Romain, > > Thanks for the reference. Do you prefer to have the Ignite runner in > Beam’s code base? > > From what I see, the current runners are hosted there: https://github.com/ > apache/beam/tree/master/runners > > As for Ignite community, we would prefer to hold the integration in your > repo. > > — > Denis > > On Dec 7, 2017, at 9:54 PM, Romain Manni-Bucau <[hidden email]> > wrote: > > Hi > > This sounds awesome to have an Ignite runner which could compete with > hazelcast-jet. > > The entry point would be https://beam.apache.org/contribute/runner-guide/ > IMHO. > > Being on Ignite cluster also opens a lot of doors - reusing the filesystem > or distributed structures. Very exiting. > > Le 8 déc. 2017 05:46, "Denis Magda" <[hidden email]> a écrit : > > Hello Apache Beam fellows! > > We at Apache Ignite community came across your project and would be happy > to integrate with it. > > In short, Ignite is a distributed database and computational platform that > has its own map-reduce like component: > https://apacheignite.readme.io/docs/compute-grid > > The integration will give Beam users an ability to use Ignite as a > distributed processing back-end system and database. > > How should we proceed? Please share any relevant information. > > — > Denis > Ignite PMC > > > |
Re: Apache Ignite as a distributed processing back-ends
|
|
Hosting integrations in impl more than beam makes a lot of sense IMHO while
you can maintain it and follow beam release cycle. It will enable you to evolve faster and optimise/adapt it more accurately. If you dont have the resources, beam would fit better and guarantee it works with each release. My 2 cts Le 13 déc. 2017 00:47, "Lukasz Cwik" <[hidden email]> a écrit : > Having it inside the Apache Beam repo makes sense and I could see it being > a good fit as an IO and as a runner. > > On Tue, Dec 12, 2017 at 3:29 PM, Denis Magda <[hidden email]> wrote: > >> Hi Romain, >> >> Thanks for the reference. Do you prefer to have the Ignite runner in >> Beam’s code base? >> >> From what I see, the current runners are hosted there: >> https://github.com/apache/beam/tree/master/runners >> >> As for Ignite community, we would prefer to hold the integration in your >> repo. >> >> — >> Denis >> >> On Dec 7, 2017, at 9:54 PM, Romain Manni-Bucau <[hidden email]> >> wrote: >> >> Hi >> >> This sounds awesome to have an Ignite runner which could compete with >> hazelcast-jet. >> >> The entry point would be https://beam.apache.org/contribute/runner-guide/ >> IMHO. >> >> Being on Ignite cluster also opens a lot of doors - reusing the filesystem >> or distributed structures. Very exiting. >> >> Le 8 déc. 2017 05:46, "Denis Magda" <[hidden email]> a écrit : >> >> Hello Apache Beam fellows! >> >> We at Apache Ignite community came across your project and would be happy >> to integrate with it. >> >> In short, Ignite is a distributed database and computational platform that >> has its own map-reduce like component: >> https://apacheignite.readme.io/docs/compute-grid >> >> The integration will give Beam users an ability to use Ignite as a >> distributed processing back-end system and database. >> >> How should we proceed? Please share any relevant information. >> >> — >> Denis >> Ignite PMC >> >> >> > |
|
|
Those are the valid points. Personally, I would go for the beam-repo way because it will guarantee that the integration works as expected with every Beam release. This is for instance how Ignite is integrated with Camel, MyBatis, Zeppelin.
Anyway, here is a ticket. Hope that an Ignite fellow will step in and solve it in the nearest time: https://issues.apache.org/jira/browse/IGNITE-7198 <https://issues.apache.org/jira/browse/IGNITE-7198> — Denis > On Dec 12, 2017, at 9:52 PM, Romain Manni-Bucau <[hidden email]> wrote: > > Hosting integrations in impl more than beam makes a lot of sense IMHO while > you can maintain it and follow beam release cycle. It will enable you to > evolve faster and optimise/adapt it more accurately. If you dont have the > resources, beam would fit better and guarantee it works with each release. > > My 2 cts > > Le 13 déc. 2017 00:47, "Lukasz Cwik" <[hidden email]> a écrit : > >> Having it inside the Apache Beam repo makes sense and I could see it being >> a good fit as an IO and as a runner. >> >> On Tue, Dec 12, 2017 at 3:29 PM, Denis Magda <[hidden email]> wrote: >> >>> Hi Romain, >>> >>> Thanks for the reference. Do you prefer to have the Ignite runner in >>> Beam’s code base? >>> >>> From what I see, the current runners are hosted there: >>> https://github.com/apache/beam/tree/master/runners >>> >>> As for Ignite community, we would prefer to hold the integration in your >>> repo. >>> >>> — >>> Denis >>> >>> On Dec 7, 2017, at 9:54 PM, Romain Manni-Bucau <[hidden email]> >>> wrote: >>> >>> Hi >>> >>> This sounds awesome to have an Ignite runner which could compete with >>> hazelcast-jet. >>> >>> The entry point would be https://beam.apache.org/contribute/runner-guide/ >>> IMHO. >>> >>> Being on Ignite cluster also opens a lot of doors - reusing the filesystem >>> or distributed structures. Very exiting. >>> >>> Le 8 déc. 2017 05:46, "Denis Magda" <[hidden email]> a écrit : >>> >>> Hello Apache Beam fellows! >>> >>> We at Apache Ignite community came across your project and would be happy >>> to integrate with it. >>> >>> In short, Ignite is a distributed database and computational platform that >>> has its own map-reduce like component: >>> https://apacheignite.readme.io/docs/compute-grid >>> >>> The integration will give Beam users an ability to use Ignite as a >>> distributed processing back-end system and database. >>> >>> How should we proceed? Please share any relevant information. >>> >>> — >>> Denis >>> Ignite PMC >>> >>> >>> >> |
«
Return to Apache Ignite Developers - Legacy Mail Archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |