IGFS

[Koert Kuipers]

"One of the unique capabilities of Ignite is a file system interface to its
in-memory data called Ignite File System (IGFS). IGFS delivers similar
functionality to Hadoop HDFS, but only in memory. In fact, IGFS is at the
core of Ignite’s In-Memory Hadoop Accelerator."

if IGFS delivers functionality similar to HDFD, then fault tolerance would
be one of the most important features. yet i cannot find any mention of it.
can someone explain? thanks!

[Konstantin Boudnik-2]

Can anyone comment on this? Seems like an interesting topic, for sure.

Thanks,
  Cos

On Sun, May 17, 2015 at 12:49AM, Koert Kuipers wrote:
> "One of the unique capabilities of Ignite is a file system interface to its
> in-memory data called Ignite File System (IGFS). IGFS delivers similar
> functionality to Hadoop HDFS, but only in memory. In fact, IGFS is at the
> core of Ignite’s In-Memory Hadoop Accelerator."
>
> if IGFS delivers functionality similar to HDFD, then fault tolerance would
> be one of the most important features. yet i cannot find any mention of it.
> can someone explain? thanks!

[Vladimir Ozerov]

Koert,

IGFS stores data in-memory. When using IGFS in pure in-memory mode, no
failover exists and data can be lost in case one of data nodes leaves the
grid.
When secondary file system is used (e.g. HDFS), then Ignite is able to
transparently re-read lost data chunks from the underlying file system.

Vladimir.

On Mon, May 25, 2015 at 1:44 PM, Konstantin Boudnik <[hidden email]> wrote:

> Can anyone comment on this? Seems like an interesting topic, for sure.
>
> Thanks,
>   Cos
>
> On Sun, May 17, 2015 at 12:49AM, Koert Kuipers wrote:
> > "One of the unique capabilities of Ignite is a file system interface to
> its
> > in-memory data called Ignite File System (IGFS). IGFS delivers similar
> > functionality to Hadoop HDFS, but only in memory. In fact, IGFS is at the
> > core of Ignite’s In-Memory Hadoop Accelerator."
> >
> > if IGFS delivers functionality similar to HDFD, then fault tolerance
> would
> > be one of the most important features. yet i cannot find any mention of
> it.
> > can someone explain? thanks!
>

[dsetrakyan]

Cos,

Which mailing list did this discussion take place in?

D.

On Mon, May 25, 2015 at 3:44 AM, Konstantin Boudnik <[hidden email]> wrote:

> Can anyone comment on this? Seems like an interesting topic, for sure.
>
> Thanks,
>   Cos
>
> On Sun, May 17, 2015 at 12:49AM, Koert Kuipers wrote:
> > "One of the unique capabilities of Ignite is a file system interface to
> its
> > in-memory data called Ignite File System (IGFS). IGFS delivers similar
> > functionality to Hadoop HDFS, but only in memory. In fact, IGFS is at the
> > core of Ignite’s In-Memory Hadoop Accelerator."
> >
> > if IGFS delivers functionality similar to HDFD, then fault tolerance
> would
> > be one of the most important features. yet i cannot find any mention of
> it.
> > can someone explain? thanks!
>

[dsetrakyan]

On Mon, May 25, 2015 at 7:18 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Koert,
>
> IGFS stores data in-memory. When using IGFS in pure in-memory mode, no
> failover exists and data can be lost in case one of data nodes leaves the
> grid.
> When secondary file system is used (e.g. HDFS), then Ignite is able to
> transparently re-read lost data chunks from the underlying file system.
>

Is this factually correct? Why can't I configure backups, just like with
regular data grids?

>
> Vladimir.
>
> On Mon, May 25, 2015 at 1:44 PM, Konstantin Boudnik <[hidden email]>
> wrote:
>
> > Can anyone comment on this? Seems like an interesting topic, for sure.
> >
> > Thanks,
> >   Cos
> >
> > On Sun, May 17, 2015 at 12:49AM, Koert Kuipers wrote:
> > > "One of the unique capabilities of Ignite is a file system interface to
> > its
> > > in-memory data called Ignite File System (IGFS). IGFS delivers similar
> > > functionality to Hadoop HDFS, but only in memory. In fact, IGFS is at
> the
> > > core of Ignite’s In-Memory Hadoop Accelerator."
> > >
> > > if IGFS delivers functionality similar to HDFD, then fault tolerance
> > would
> > > be one of the most important features. yet i cannot find any mention of
> > it.
> > > can someone explain? thanks!
> >
>

[Branko Čibej]

On 25.05.2015 17:52, Dmitriy Setrakyan wrote:
> Cos,
>
> Which mailing list did this discussion take place in?

This one.

http://mail-archives.apache.org/mod_mbox/incubator-ignite-dev/201505.mbox/%3CCANx3uAg2J9qvj2mP2MsnsXd1A4F6c1zgpOm4iuQ8WzEzi-Uywg%40mail.gmail.com%3E

-- Brane

[Vladimir Ozerov]

I already raised question about backups several months ago but no
implementation decisions were made.
I am not aware of any architectural problems preventing us form having
backups, but currently backups is prohibited eplicitly, and user will get
validation error if backup is configured on data cache. However, this makes
sense in the most common use case - IGFS as a caching layer over some
perstent file system.

On Mon, May 25, 2015 at 7:07 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Mon, May 25, 2015 at 7:18 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Koert,
> >
> > IGFS stores data in-memory. When using IGFS in pure in-memory mode, no
> > failover exists and data can be lost in case one of data nodes leaves the
> > grid.
> > When secondary file system is used (e.g. HDFS), then Ignite is able to
> > transparently re-read lost data chunks from the underlying file system.
> >
>
> Is this factually correct? Why can't I configure backups, just like with
> regular data grids?
>
>
> >
> > Vladimir.
> >
> > On Mon, May 25, 2015 at 1:44 PM, Konstantin Boudnik <[hidden email]>
> > wrote:
> >
> > > Can anyone comment on this? Seems like an interesting topic, for sure.
> > >
> > > Thanks,
> > >   Cos
> > >
> > > On Sun, May 17, 2015 at 12:49AM, Koert Kuipers wrote:
> > > > "One of the unique capabilities of Ignite is a file system interface
> to
> > > its
> > > > in-memory data called Ignite File System (IGFS). IGFS delivers
> similar
> > > > functionality to Hadoop HDFS, but only in memory. In fact, IGFS is at
> > the
> > > > core of Ignite’s In-Memory Hadoop Accelerator."
> > > >
> > > > if IGFS delivers functionality similar to HDFD, then fault tolerance
> > > would
> > > > be one of the most important features. yet i cannot find any mention
> of
> > > it.
> > > > can someone explain? thanks!
> > >
> >
>

[dsetrakyan]

On Mon, May 25, 2015 at 11:04 AM, Vladimir Ozerov <[hidden email]>
wrote:

> I already raised question about backups several months ago but no
> implementation decisions were made.
> I am not aware of any architectural problems preventing us form having
> backups, but currently backups is prohibited eplicitly, and user will get
> validation error if backup is configured on data cache. However, this makes
> sense in the most common use case - IGFS as a caching layer over some
> perstent file system.
>

I would vote to remove such restriction. I think the rationale before was
to prevent users from allocating extra memory for backups when they could
simply persist to HDFS. But now we are seeing that people are starting to
use IGFS without Hadoop altogether.

Alexey G, can you share your thoughts on the matter as well?

>
> On Mon, May 25, 2015 at 7:07 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Mon, May 25, 2015 at 7:18 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Koert,
> > >
> > > IGFS stores data in-memory. When using IGFS in pure in-memory mode, no
> > > failover exists and data can be lost in case one of data nodes leaves
> the
> > > grid.
> > > When secondary file system is used (e.g. HDFS), then Ignite is able to
> > > transparently re-read lost data chunks from the underlying file system.
> > >
> >
> > Is this factually correct? Why can't I configure backups, just like with
> > regular data grids?
> >
> >
> > >
> > > Vladimir.
> > >
> > > On Mon, May 25, 2015 at 1:44 PM, Konstantin Boudnik <[hidden email]>
> > > wrote:
> > >
> > > > Can anyone comment on this? Seems like an interesting topic, for
> sure.
> > > >
> > > > Thanks,
> > > >   Cos
> > > >
> > > > On Sun, May 17, 2015 at 12:49AM, Koert Kuipers wrote:
> > > > > "One of the unique capabilities of Ignite is a file system
> interface
> > to
> > > > its
> > > > > in-memory data called Ignite File System (IGFS). IGFS delivers
> > similar
> > > > > functionality to Hadoop HDFS, but only in memory. In fact, IGFS is
> at
> > > the
> > > > > core of Ignite’s In-Memory Hadoop Accelerator."
> > > > >
> > > > > if IGFS delivers functionality similar to HDFD, then fault
> tolerance
> > > > would
> > > > > be one of the most important features. yet i cannot find any
> mention
> > of
> > > > it.
> > > > > can someone explain? thanks!
> > > >
> > >
> >
>

[Branko Čibej]

On 25.05.2015 20:04, Vladimir Ozerov wrote:
> I already raised question about backups several months ago but no
> implementation decisions were made.
> I am not aware of any architectural problems preventing us form having
> backups, but currently backups is prohibited eplicitly, and user will get
> validation error if backup is configured on data cache. However, this makes
> sense in the most common use case - IGFS as a caching layer over some
> perstent file system.


You know, you don't actually have to wait for anyone's approval to
implement a feature.

Open-source volunteer, remember? Not day-job kowtow-to-boss. :)

-- Brane