Ceph w/ Ignite

[Konstantin Boudnik-2]

Guys,

I was looking at the Hadoop accelerator the other day and been thinking if
anyone has tried to use IGFS on top of a real distributed file storage. The
case in point is Ceph (ceph.com) - a Linux file system available from any
major Linux distribution as a kernel module.

HDFS has its share in the world, but it isn't the fastest, simplest, nor most
advantageous distributed storage on the planet. Hence I am wondering if this
would be a good call to provide Ignite on CEPH as a 2nd FS capabilities.

Thoughts?
  Cos

[Roman Shaposhnik]

I'm sure our RH brethren have something to say about Ceph.
Re-post on dev@bigtop?

Thanks,
Roman.

On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]> wrote:

> Guys,
>
> I was looking at the Hadoop accelerator the other day and been thinking if
> anyone has tried to use IGFS on top of a real distributed file storage. The
> case in point is Ceph (ceph.com) - a Linux file system available from any
> major Linux distribution as a kernel module.
>
> HDFS has its share in the world, but it isn't the fastest, simplest, nor most
> advantageous distributed storage on the planet. Hence I am wondering if this
> would be a good call to provide Ignite on CEPH as a 2nd FS capabilities.
>
> Thoughts?
>   Cos
>

[Konstantin Boudnik-2]

Good point... although I was curious about Ignite's take on that first and
foremost. Yet, cross-posting to [hidden email]

Jay et all: any thoughts about the combination?
  Cos

On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:

> I'm sure our RH brethren have something to say about Ceph.
> Re-post on dev@bigtop?
>
> Thanks,
> Roman.
>
> On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]> wrote:
> > Guys,
> >
> > I was looking at the Hadoop accelerator the other day and been thinking if
> > anyone has tried to use IGFS on top of a real distributed file storage. The
> > case in point is Ceph (ceph.com) - a Linux file system available from any
> > major Linux distribution as a kernel module.
> >
> > HDFS has its share in the world, but it isn't the fastest, simplest, nor most
> > advantageous distributed storage on the planet. Hence I am wondering if this
> > would be a good call to provide Ignite on CEPH as a 2nd FS capabilities.
> >
> > Thoughts?
> >   Cos
> >

[RJ Nowling]

Ceph makes a better object store while Gluster makes a better file system.
  That's why Ceph is a popular backend for OpenStack Swift.

Does Ignite want a FS or Object backend?

On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]> wrote:

> Good point... although I was curious about Ignite's take on that first and
> foremost. Yet, cross-posting to [hidden email]
>
> Jay et all: any thoughts about the combination?
>   Cos
>
> On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > I'm sure our RH brethren have something to say about Ceph.
> > Re-post on dev@bigtop?
> >
> > Thanks,
> > Roman.
> >
> > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> > > Guys,
> > >
> > > I was looking at the Hadoop accelerator the other day and been
> thinking if
> > > anyone has tried to use IGFS on top of a real distributed file
> storage. The
> > > case in point is Ceph (ceph.com) - a Linux file system available from
> any
> > > major Linux distribution as a kernel module.
> > >
> > > HDFS has its share in the world, but it isn't the fastest, simplest,
> nor most
> > > advantageous distributed storage on the planet. Hence I am wondering
> if this
> > > would be a good call to provide Ignite on CEPH as a 2nd FS
> capabilities.
> > >
> > > Thoughts?
> > >   Cos
> > >
>

[jay vyas]

in particular, w/ ignite, i could see web apps w/ large object stores as
storage, doing larger, faster
queries for certain workloads, i.e. the using the jdbc stuff. same thing
for in house apps that might
need to operate on large posix stored data (like in gluster)




On Wed, Jul 8, 2015 at 11:40 PM, RJ Nowling <[hidden email]> wrote:

> Ceph makes a better object store while Gluster makes a better file system.
>   That's why Ceph is a popular backend for OpenStack Swift.
>
> Does Ignite want a FS or Object backend?
>
> On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> > Good point... although I was curious about Ignite's take on that first
> and
> > foremost. Yet, cross-posting to [hidden email]
> >
> > Jay et all: any thoughts about the combination?
> >   Cos
> >
> > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > I'm sure our RH brethren have something to say about Ceph.
> > > Re-post on dev@bigtop?
> > >
> > > Thanks,
> > > Roman.
> > >
> > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]>
> > wrote:
> > > > Guys,
> > > >
> > > > I was looking at the Hadoop accelerator the other day and been
> > thinking if
> > > > anyone has tried to use IGFS on top of a real distributed file
> > storage. The
> > > > case in point is Ceph (ceph.com) - a Linux file system available
> from
> > any
> > > > major Linux distribution as a kernel module.
> > > >
> > > > HDFS has its share in the world, but it isn't the fastest, simplest,
> > nor most
> > > > advantageous distributed storage on the planet. Hence I am wondering
> > if this
> > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > capabilities.
> > > >
> > > > Thoughts?
> > > >   Cos
> > > >
> >
>

--
jay vyas

[dsetrakyan]

Hm... I would think that file system would be more beneficial, although
object store on disk can also be valuable.

Cos, what is your thinking?

D.

On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]> wrote:

> Ceph makes a better object store while Gluster makes a better file system.
>   That's why Ceph is a popular backend for OpenStack Swift.
>
> Does Ignite want a FS or Object backend?
>
> On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> > Good point... although I was curious about Ignite's take on that first
> and
> > foremost. Yet, cross-posting to [hidden email]
> >
> > Jay et all: any thoughts about the combination?
> >   Cos
> >
> > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > I'm sure our RH brethren have something to say about Ceph.
> > > Re-post on dev@bigtop?
> > >
> > > Thanks,
> > > Roman.
> > >
> > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]>
> > wrote:
> > > > Guys,
> > > >
> > > > I was looking at the Hadoop accelerator the other day and been
> > thinking if
> > > > anyone has tried to use IGFS on top of a real distributed file
> > storage. The
> > > > case in point is Ceph (ceph.com) - a Linux file system available
> from
> > any
> > > > major Linux distribution as a kernel module.
> > > >
> > > > HDFS has its share in the world, but it isn't the fastest, simplest,
> > nor most
> > > > advantageous distributed storage on the planet. Hence I am wondering
> > if this
> > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > capabilities.
> > > >
> > > > Thoughts?
> > > >   Cos
> > > >
> >
>

[Atri Sharma]

Object store is *very* valuable for some use cases I know. Especially,
using Ignite as a high speed in memory object processor is good for some
warehousing projects.

Also, I think implementing features like Write Ahead Logging really
simplify with a persistent Object store.

On Thu, Jul 9, 2015 at 2:23 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Hm... I would think that file system would be more beneficial, although
> object store on disk can also be valuable.
>
> Cos, what is your thinking?
>
> D.
>
> On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]> wrote:
>
> > Ceph makes a better object store while Gluster makes a better file
> system.
> >   That's why Ceph is a popular backend for OpenStack Swift.
> >
> > Does Ignite want a FS or Object backend?
> >
> > On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> >
> > > Good point... although I was curious about Ignite's take on that first
> > and
> > > foremost. Yet, cross-posting to [hidden email]
> > >
> > > Jay et all: any thoughts about the combination?
> > >   Cos
> > >
> > > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > > I'm sure our RH brethren have something to say about Ceph.
> > > > Re-post on dev@bigtop?
> > > >
> > > > Thanks,
> > > > Roman.
> > > >
> > > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]>
> > > wrote:
> > > > > Guys,
> > > > >
> > > > > I was looking at the Hadoop accelerator the other day and been
> > > thinking if
> > > > > anyone has tried to use IGFS on top of a real distributed file
> > > storage. The
> > > > > case in point is Ceph (ceph.com) - a Linux file system available
> > from
> > > any
> > > > > major Linux distribution as a kernel module.
> > > > >
> > > > > HDFS has its share in the world, but it isn't the fastest,
> simplest,
> > > nor most
> > > > > advantageous distributed storage on the planet. Hence I am
> wondering
> > > if this
> > > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > > capabilities.
> > > > >
> > > > > Thoughts?
> > > > >   Cos
> > > > >
> > >
> >
>

--
Regards,

Atri
*l'apprenant*

[Konstantin Boudnik-2]

I think file system is more universally used. However, one can build an FS on
top of a good object storage - just need to provide some metadata
abstraction/concept.

Replacing HDFS w/ Gluster doesn't make much sense to me (if ever be
considered). What I like about Ceph is that it is native to Linux, unlike all
other artificial HCFS contraptions. Hence my initial question.

Cos

On Thu, Jul 09, 2015 at 01:53AM, Dmitriy Setrakyan wrote:

> Hm... I would think that file system would be more beneficial, although
> object store on disk can also be valuable.
>
> Cos, what is your thinking?
>
> D.
>
> On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]> wrote:
>
> > Ceph makes a better object store while Gluster makes a better file system.
> >   That's why Ceph is a popular backend for OpenStack Swift.
> >
> > Does Ignite want a FS or Object backend?
> >
> > On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]> wrote:
> >
> > > Good point... although I was curious about Ignite's take on that first
> > and
> > > foremost. Yet, cross-posting to [hidden email]
> > >
> > > Jay et all: any thoughts about the combination?
> > >   Cos
> > >
> > > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > > I'm sure our RH brethren have something to say about Ceph.
> > > > Re-post on dev@bigtop?
> > > >
> > > > Thanks,
> > > > Roman.
> > > >
> > > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]>
> > > wrote:
> > > > > Guys,
> > > > >
> > > > > I was looking at the Hadoop accelerator the other day and been
> > > thinking if
> > > > > anyone has tried to use IGFS on top of a real distributed file
> > > storage. The
> > > > > case in point is Ceph (ceph.com) - a Linux file system available
> > from
> > > any
> > > > > major Linux distribution as a kernel module.
> > > > >
> > > > > HDFS has its share in the world, but it isn't the fastest, simplest,
> > > nor most
> > > > > advantageous distributed storage on the planet. Hence I am wondering
> > > if this
> > > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > > capabilities.
> > > > >
> > > > > Thoughts?
> > > > >   Cos
> > > > >
> > >
> >

[RJ Nowling]

Cos,

Can you expand on what you mean by "native to Linux" for Ceph?

And can you elaborate on why Gluster doesn't make sense as a HDFS
replacement to you?

Not trying to argue -- just generally curious.  :)

Thanks!

On Mon, Jul 13, 2015 at 5:06 PM, Konstantin Boudnik <[hidden email]> wrote:

> I think file system is more universally used. However, one can build an FS
> on
> top of a good object storage - just need to provide some metadata
> abstraction/concept.
>
> Replacing HDFS w/ Gluster doesn't make much sense to me (if ever be
> considered). What I like about Ceph is that it is native to Linux, unlike
> all
> other artificial HCFS contraptions. Hence my initial question.
>
> Cos
>
> On Thu, Jul 09, 2015 at 01:53AM, Dmitriy Setrakyan wrote:
> > Hm... I would think that file system would be more beneficial, although
> > object store on disk can also be valuable.
> >
> > Cos, what is your thinking?
> >
> > D.
> >
> > On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]> wrote:
> >
> > > Ceph makes a better object store while Gluster makes a better file
> system.
> > >   That's why Ceph is a popular backend for OpenStack Swift.
> > >
> > > Does Ignite want a FS or Object backend?
> > >
> > > On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> > >
> > > > Good point... although I was curious about Ignite's take on that
> first
> > > and
> > > > foremost. Yet, cross-posting to [hidden email]
> > > >
> > > > Jay et all: any thoughts about the combination?
> > > >   Cos
> > > >
> > > > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > > > I'm sure our RH brethren have something to say about Ceph.
> > > > > Re-post on dev@bigtop?
> > > > >
> > > > > Thanks,
> > > > > Roman.
> > > > >
> > > > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]
> >
> > > > wrote:
> > > > > > Guys,
> > > > > >
> > > > > > I was looking at the Hadoop accelerator the other day and been
> > > > thinking if
> > > > > > anyone has tried to use IGFS on top of a real distributed file
> > > > storage. The
> > > > > > case in point is Ceph (ceph.com) - a Linux file system available
> > > from
> > > > any
> > > > > > major Linux distribution as a kernel module.
> > > > > >
> > > > > > HDFS has its share in the world, but it isn't the fastest,
> simplest,
> > > > nor most
> > > > > > advantageous distributed storage on the planet. Hence I am
> wondering
> > > > if this
> > > > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > > > capabilities.
> > > > > >
> > > > > > Thoughts?
> > > > > >   Cos
> > > > > >
> > > >
> > >
>

[Konstantin Boudnik-2]

On Mon, Jul 13, 2015 at 07:00PM, RJ Nowling wrote:
> Cos,
>
> Can you expand on what you mean by "native to Linux" for Ceph?

I meant that the file system is presented in a Linux distro as kernel module.
HDFS, as you know, is an alien Java process that creates a layer indirection
on top of say ext4 or jfs to provide a distributed storage; Ceph does this
similarly to other _native_ file systems.

> And can you elaborate on why Gluster doesn't make sense as a HDFS
> replacement to you?

What I wanted to express, perhaps a bit clumsy, is that HDFS and Gluster are
two instances of HCFS. from Ignite standpoint replacing one with another
doesn't give much advantage (unless I am missing something about the Gluster).
Hopefully it makes sense?

> Not trying to argue -- just generally curious.  :)

Not trying to cast a shadow on Gluster nor whitewash HDFS (far from it) ;)

Cos

> Thanks!
>
> On Mon, Jul 13, 2015 at 5:06 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> > I think file system is more universally used. However, one can build an FS
> > on
> > top of a good object storage - just need to provide some metadata
> > abstraction/concept.
> >
> > Replacing HDFS w/ Gluster doesn't make much sense to me (if ever be
> > considered). What I like about Ceph is that it is native to Linux, unlike
> > all
> > other artificial HCFS contraptions. Hence my initial question.
> >
> > Cos
> >
> > On Thu, Jul 09, 2015 at 01:53AM, Dmitriy Setrakyan wrote:
> > > Hm... I would think that file system would be more beneficial, although
> > > object store on disk can also be valuable.
> > >
> > > Cos, what is your thinking?
> > >
> > > D.
> > >
> > > On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]> wrote:
> > >
> > > > Ceph makes a better object store while Gluster makes a better file
> > system.
> > > >   That's why Ceph is a popular backend for OpenStack Swift.
> > > >
> > > > Does Ignite want a FS or Object backend?
> > > >
> > > > On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]>
> > wrote:
> > > >
> > > > > Good point... although I was curious about Ignite's take on that
> > first
> > > > and
> > > > > foremost. Yet, cross-posting to [hidden email]
> > > > >
> > > > > Jay et all: any thoughts about the combination?
> > > > >   Cos
> > > > >
> > > > > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > > > > I'm sure our RH brethren have something to say about Ceph.
> > > > > > Re-post on dev@bigtop?
> > > > > >
> > > > > > Thanks,
> > > > > > Roman.
> > > > > >
> > > > > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <[hidden email]
> > >
> > > > > wrote:
> > > > > > > Guys,
> > > > > > >
> > > > > > > I was looking at the Hadoop accelerator the other day and been
> > > > > thinking if
> > > > > > > anyone has tried to use IGFS on top of a real distributed file
> > > > > storage. The
> > > > > > > case in point is Ceph (ceph.com) - a Linux file system available
> > > > from
> > > > > any
> > > > > > > major Linux distribution as a kernel module.
> > > > > > >
> > > > > > > HDFS has its share in the world, but it isn't the fastest,
> > simplest,
> > > > > nor most
> > > > > > > advantageous distributed storage on the planet. Hence I am
> > wondering
> > > > > if this
> > > > > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > > > > capabilities.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >   Cos
> > > > > > >
> > > > >
> > > >
> >

[RJ Nowling]

Thanks, Cos!

> from Ignite standpoint replacing one with another doesn't give much
advantage

Agreed.  From the standpoint of Ignite, Hadoop, or Spark, Gluster works no
differently than HDFS.  If Ignite doesn't have an object store available
already, then Ceph could add that capability.

From the standpoint of the user and integration with a larger IT
infrastructure, Gluster offers advantages over HDFS.  As you say, Gluster
is a POSIX-compatible native filesystem -- it provides a FUSE module for
mounting remote Gluster volumes.  This means non-Hadoop applications can
store data in the same file system as Hadoop.

I come from a scientific computing background where pretty much every
simulation or analysis tool expected access to a POSIX file system.  We
evaluated Hadoop at one point but chose not to use it because we would have
to copy all of our data into HDFS.  Gluster is a much better POSIX
distributed file system than what my university's cluster used, and I wish
I had known about it while doing my Ph.D.  :)

For my work at Red Hat, we run Spark on Gluster.  We don't use any special
plugins -- since Spark uses the Hadoop file system libraries, Spark can
read off native file systems.  Same advantages mentioned above -- nice to
be able to use grep, cat, etc. alongside Spark :)


On Mon, Jul 13, 2015 at 8:55 PM, Konstantin Boudnik <[hidden email]> wrote:

> On Mon, Jul 13, 2015 at 07:00PM, RJ Nowling wrote:
> > Cos,
> >
> > Can you expand on what you mean by "native to Linux" for Ceph?
>
> I meant that the file system is presented in a Linux distro as kernel
> module.
> HDFS, as you know, is an alien Java process that creates a layer
> indirection
> on top of say ext4 or jfs to provide a distributed storage; Ceph does this
> similarly to other _native_ file systems.
>
> > And can you elaborate on why Gluster doesn't make sense as a HDFS
> > replacement to you?
>
> What I wanted to express, perhaps a bit clumsy, is that HDFS and Gluster
> are
> two instances of HCFS. from Ignite standpoint replacing one with another
> doesn't give much advantage (unless I am missing something about the
> Gluster).
> Hopefully it makes sense?
>
> > Not trying to argue -- just generally curious.  :)
>
> Not trying to cast a shadow on Gluster nor whitewash HDFS (far from it) ;)
>
> Cos
>
> > Thanks!
> >
> > On Mon, Jul 13, 2015 at 5:06 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> >
> > > I think file system is more universally used. However, one can build
> an FS
> > > on
> > > top of a good object storage - just need to provide some metadata
> > > abstraction/concept.
> > >
> > > Replacing HDFS w/ Gluster doesn't make much sense to me (if ever be
> > > considered). What I like about Ceph is that it is native to Linux,
> unlike
> > > all
> > > other artificial HCFS contraptions. Hence my initial question.
> > >
> > > Cos
> > >
> > > On Thu, Jul 09, 2015 at 01:53AM, Dmitriy Setrakyan wrote:
> > > > Hm... I would think that file system would be more beneficial,
> although
> > > > object store on disk can also be valuable.
> > > >
> > > > Cos, what is your thinking?
> > > >
> > > > D.
> > > >
> > > > On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]>
> wrote:
> > > >
> > > > > Ceph makes a better object store while Gluster makes a better file
> > > system.
> > > > >   That's why Ceph is a popular backend for OpenStack Swift.
> > > > >
> > > > > Does Ignite want a FS or Object backend?
> > > > >
> > > > > On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik <[hidden email]
> >
> > > wrote:
> > > > >
> > > > > > Good point... although I was curious about Ignite's take on that
> > > first
> > > > > and
> > > > > > foremost. Yet, cross-posting to [hidden email]
> > > > > >
> > > > > > Jay et all: any thoughts about the combination?
> > > > > >   Cos
> > > > > >
> > > > > > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
> > > > > > > I'm sure our RH brethren have something to say about Ceph.
> > > > > > > Re-post on dev@bigtop?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Roman.
> > > > > > >
> > > > > > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <
> [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > I was looking at the Hadoop accelerator the other day and
> been
> > > > > > thinking if
> > > > > > > > anyone has tried to use IGFS on top of a real distributed
> file
> > > > > > storage. The
> > > > > > > > case in point is Ceph (ceph.com) - a Linux file system
> available
> > > > > from
> > > > > > any
> > > > > > > > major Linux distribution as a kernel module.
> > > > > > > >
> > > > > > > > HDFS has its share in the world, but it isn't the fastest,
> > > simplest,
> > > > > > nor most
> > > > > > > > advantageous distributed storage on the planet. Hence I am
> > > wondering
> > > > > > if this
> > > > > > > > would be a good call to provide Ignite on CEPH as a 2nd FS
> > > > > > capabilities.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >   Cos
> > > > > > > >
> > > > > >
> > > > >
> > >
>

[Konstantin Boudnik-2]

All arw good points! Thanks for the details about Gluster - surely helpful.

On July 13, 2015 9:23:33 PM PDT, RJ Nowling <[hidden email]> wrote:

>Thanks, Cos!
>
>> from Ignite standpoint replacing one with another doesn't give much
>advantage
>
>Agreed.  From the standpoint of Ignite, Hadoop, or Spark, Gluster works
>no
>differently than HDFS.  If Ignite doesn't have an object store
>available
>already, then Ceph could add that capability.
>
>From the standpoint of the user and integration with a larger IT
>infrastructure, Gluster offers advantages over HDFS.  As you say,
>Gluster
>is a POSIX-compatible native filesystem -- it provides a FUSE module
>for
>mounting remote Gluster volumes.  This means non-Hadoop applications
>can
>store data in the same file system as Hadoop.
>
>I come from a scientific computing background where pretty much every
>simulation or analysis tool expected access to a POSIX file system.  We
>evaluated Hadoop at one point but chose not to use it because we would
>have
>to copy all of our data into HDFS.  Gluster is a much better POSIX
>distributed file system than what my university's cluster used, and I
>wish
>I had known about it while doing my Ph.D.  :)
>
>For my work at Red Hat, we run Spark on Gluster.  We don't use any
>special
>plugins -- since Spark uses the Hadoop file system libraries, Spark can
>read off native file systems.  Same advantages mentioned above -- nice
>to
>be able to use grep, cat, etc. alongside Spark :)
>
>
>On Mon, Jul 13, 2015 at 8:55 PM, Konstantin Boudnik <[hidden email]>
>wrote:
>
>> On Mon, Jul 13, 2015 at 07:00PM, RJ Nowling wrote:
>> > Cos,
>> >
>> > Can you expand on what you mean by "native to Linux" for Ceph?
>>
>> I meant that the file system is presented in a Linux distro as kernel
>> module.
>> HDFS, as you know, is an alien Java process that creates a layer
>> indirection
>> on top of say ext4 or jfs to provide a distributed storage; Ceph does
>this
>> similarly to other _native_ file systems.
>>
>> > And can you elaborate on why Gluster doesn't make sense as a HDFS
>> > replacement to you?
>>
>> What I wanted to express, perhaps a bit clumsy, is that HDFS and
>Gluster
>> are
>> two instances of HCFS. from Ignite standpoint replacing one with
>another
>> doesn't give much advantage (unless I am missing something about the
>> Gluster).
>> Hopefully it makes sense?
>>
>> > Not trying to argue -- just generally curious.  :)
>>
>> Not trying to cast a shadow on Gluster nor whitewash HDFS (far from
>it) ;)
>>
>> Cos
>>
>> > Thanks!
>> >
>> > On Mon, Jul 13, 2015 at 5:06 PM, Konstantin Boudnik
><[hidden email]>
>> wrote:
>> >
>> > > I think file system is more universally used. However, one can
>build
>> an FS
>> > > on
>> > > top of a good object storage - just need to provide some metadata
>> > > abstraction/concept.
>> > >
>> > > Replacing HDFS w/ Gluster doesn't make much sense to me (if ever
>be
>> > > considered). What I like about Ceph is that it is native to
>Linux,
>> unlike
>> > > all
>> > > other artificial HCFS contraptions. Hence my initial question.
>> > >
>> > > Cos
>> > >
>> > > On Thu, Jul 09, 2015 at 01:53AM, Dmitriy Setrakyan wrote:
>> > > > Hm... I would think that file system would be more beneficial,
>> although
>> > > > object store on disk can also be valuable.
>> > > >
>> > > > Cos, what is your thinking?
>> > > >
>> > > > D.
>> > > >
>> > > > On Wed, Jul 8, 2015 at 8:40 PM, RJ Nowling <[hidden email]>
>> wrote:
>> > > >
>> > > > > Ceph makes a better object store while Gluster makes a better
>file
>> > > system.
>> > > > >   That's why Ceph is a popular backend for OpenStack Swift.
>> > > > >
>> > > > > Does Ignite want a FS or Object backend?
>> > > > >
>> > > > > On Wed, Jul 8, 2015 at 5:57 PM, Konstantin Boudnik
><[hidden email]
>> >
>> > > wrote:
>> > > > >
>> > > > > > Good point... although I was curious about Ignite's take on
>that
>> > > first
>> > > > > and
>> > > > > > foremost. Yet, cross-posting to [hidden email]
>> > > > > >
>> > > > > > Jay et all: any thoughts about the combination?
>> > > > > >   Cos
>> > > > > >
>> > > > > > On Wed, Jul 08, 2015 at 03:14PM, Roman Shaposhnik wrote:
>> > > > > > > I'm sure our RH brethren have something to say about
>Ceph.
>> > > > > > > Re-post on dev@bigtop?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Roman.
>> > > > > > >
>> > > > > > > On Wed, Jul 8, 2015 at 3:17 PM, Konstantin Boudnik <
>> [hidden email]
>> > > >
>> > > > > > wrote:
>> > > > > > > > Guys,
>> > > > > > > >
>> > > > > > > > I was looking at the Hadoop accelerator the other day
>and
>> been
>> > > > > > thinking if
>> > > > > > > > anyone has tried to use IGFS on top of a real
>distributed
>> file
>> > > > > > storage. The
>> > > > > > > > case in point is Ceph (ceph.com) - a Linux file system
>> available
>> > > > > from
>> > > > > > any
>> > > > > > > > major Linux distribution as a kernel module.
>> > > > > > > >
>> > > > > > > > HDFS has its share in the world, but it isn't the
>fastest,
>> > > simplest,
>> > > > > > nor most
>> > > > > > > > advantageous distributed storage on the planet. Hence I
>am
>> > > wondering
>> > > > > > if this
>> > > > > > > > would be a good call to provide Ignite on CEPH as a 2nd
>FS
>> > > > > > capabilities.
>> > > > > > > >
>> > > > > > > > Thoughts?
>> > > > > > > >   Cos
>> > > > > > > >
>> > > > > >
>> > > > >
>> > >
>>