Integrating Apache Ignite with Mesos

[dsetrakyan]

Hello Mesos community,

I would like to start a conversation on integrating Apache Ignite with
Mesos. If you are not familiar with Ignite, it is an in-memory data fabric
providing clustering and various distributed caching and data processing
capabilities. More on Ignite here: https://ignite.incubator.apache.org/

To get started, we are looking at various existing implementations. Do you
have examples of distributed in memory systems running on top of Mesos? We
have looked at Spark and will probably take it as a reference
implementation, but if you can point us to other examples, it would be
great.

Also, is there a common place to host the integration modules for Mesos? I
have done some searching but could not find anything.

Thanks,
D.

[dsetrakyan]

On Sun, May 10, 2015 at 10:16 AM, haosdent <[hidden email]> wrote:

> How about tachyon? https://github.com/mesosphere/tachyon-mesos I just
> search it from github.
>

Thanks, this is helpful. Ignite actually also has a hadoop-compliant
in-memory file system, so we may be adding 2 integrations, one for the
in-memory data fabric, and another for the file system.

Is mesosphere github repository a general place where such integrations are
hosted? Or is there a special Apache repository for Apache projects?

>
> On Mon, May 11, 2015 at 12:34 AM, Dmitriy Setrakyan <[hidden email]
> >
> wrote:
>
> > Hello Mesos community,
> >
> > I would like to start a conversation on integrating Apache Ignite with
> > Mesos. If you are not familiar with Ignite, it is an in-memory data
> fabric
> > providing clustering and various distributed caching and data processing
> > capabilities. More on Ignite here: https://ignite.incubator.apache.org/
> >
> > To get started, we are looking at various existing implementations. Do
> you
> > have examples of distributed in memory systems running on top of Mesos?
> We
> > have looked at Spark and will probably take it as a reference
> > implementation, but if you can point us to other examples, it would be
> > great.
> >
> > Also, is there a common place to host the integration modules for Mesos?
> I
> > have done some searching but could not find anything.
> >
> > Thanks,
> > D.
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

[Vinod Kone]

Mesos frameworks are typically hosted under github/mesos.

@vinodkone

> On May 10, 2015, at 3:40 PM, Dmitriy Setrakyan <[hidden email]> wrote:
>
>> On Sun, May 10, 2015 at 10:16 AM, haosdent <[hidden email]> wrote:
>>
>> How about tachyon? https://github.com/mesosphere/tachyon-mesos I just
>> search it from github.
>
> Thanks, this is helpful. Ignite actually also has a hadoop-compliant
> in-memory file system, so we may be adding 2 integrations, one for the
> in-memory data fabric, and another for the file system.
>
> Is mesosphere github repository a general place where such integrations are
> hosted? Or is there a special Apache repository for Apache projects?
>
>
>>
>> On Mon, May 11, 2015 at 12:34 AM, Dmitriy Setrakyan <[hidden email]
>> wrote:
>>
>>> Hello Mesos community,
>>>
>>> I would like to start a conversation on integrating Apache Ignite with
>>> Mesos. If you are not familiar with Ignite, it is an in-memory data
>> fabric
>>> providing clustering and various distributed caching and data processing
>>> capabilities. More on Ignite here: https://ignite.incubator.apache.org/
>>>
>>> To get started, we are looking at various existing implementations. Do
>> you
>>> have examples of distributed in memory systems running on top of Mesos?
>> We
>>> have looked at Spark and will probably take it as a reference
>>> implementation, but if you can point us to other examples, it would be
>>> great.
>>>
>>> Also, is there a common place to host the integration modules for Mesos?
>> I
>>> have done some searching but could not find anything.
>>>
>>> Thanks,
>>> D.
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>

[Adam Bordelon]

The github.com/mesosphere/ org is for projects under the direction of the
company Mesosphere. Probably not the right place unless you want to donate
the code to Mesosphere.

github.com/mesos/ is the community repo, administered by Apache Mesos
committers, but not under the direct jurisdiction of Apache Software
Foundation itself. This is a great starting place for community projects,
although some (e.g. Apache Spark <https://github.com/mesos/spark> or Apache
Myriad <https://github.com/mesos/myriad> for YARN) eventually become their
own Apache projects. This may be a suitable place for your framework to
start. Let us know and we can fork an existing repo or create a new one for
you.

Alternatively, you may wish to push the Ignite-Mesos framework upstream to
the Apache Ignite codebase itself. Apache Spark includes the Mesos
framework in its main codebase. The Kubernetes-Mesos framework is working
on pushing the framework code upstream to Kubernetes proper. For this to
work, you will need buy-in from the Ignite community.

On Sun, May 10, 2015 at 4:14 PM, Vinod Kone <[hidden email]> wrote:

> Mesos frameworks are typically hosted under github/mesos.
>
> @vinodkone
>
> > On May 10, 2015, at 3:40 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
> >
> >> On Sun, May 10, 2015 at 10:16 AM, haosdent <[hidden email]> wrote:
> >>
> >> How about tachyon? https://github.com/mesosphere/tachyon-mesos I just
> >> search it from github.
> >
> > Thanks, this is helpful. Ignite actually also has a hadoop-compliant
> > in-memory file system, so we may be adding 2 integrations, one for the
> > in-memory data fabric, and another for the file system.
> >
> > Is mesosphere github repository a general place where such integrations
> are
> > hosted? Or is there a special Apache repository for Apache projects?
> >
> >
> >>
> >> On Mon, May 11, 2015 at 12:34 AM, Dmitriy Setrakyan <
> [hidden email]
> >> wrote:
> >>
> >>> Hello Mesos community,
> >>>
> >>> I would like to start a conversation on integrating Apache Ignite with
> >>> Mesos. If you are not familiar with Ignite, it is an in-memory data
> >> fabric
> >>> providing clustering and various distributed caching and data
> processing
> >>> capabilities. More on Ignite here:
> https://ignite.incubator.apache.org/
> >>>
> >>> To get started, we are looking at various existing implementations. Do
> >> you
> >>> have examples of distributed in memory systems running on top of Mesos?
> >> We
> >>> have looked at Spark and will probably take it as a reference
> >>> implementation, but if you can point us to other examples, it would be
> >>> great.
> >>>
> >>> Also, is there a common place to host the integration modules for
> Mesos?
> >> I
> >>> have done some searching but could not find anything.
> >>>
> >>> Thanks,
> >>> D.
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Haosdent Huang
> >>
>