Ignite Direct I/O plugin description added to wiki

[Dmitriy Pavlov]

Hi Igniters,

I've added description of new plugin for Direct I/O for native persistence (
https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-DirectI/O


SIncerely,
Dmitriy Pavlov

[dsetrakyan]

Thanks Dmitry, awesome work!

On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Igniters,
>
> I've added description of new plugin for Direct I/O for native persistence
> (
> https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> underthehood-DirectI/O
>
>
> SIncerely,
> Dmitriy Pavlov
>

[Raymond Wilson]

Looks good!

Is there any reason why this should not be a default setting if it
gracefully downgrades to non-Direct IO if not supported by the OS?

Thanks,
Raymond.

-----Original Message-----
From: Dmitriy Setrakyan [mailto:[hidden email]]
Sent: Wednesday, March 21, 2018 10:23 AM
To: dev <[hidden email]>
Subject: Re: Ignite Direct I/O plugin description added to wiki

Thanks Dmitry, awesome work!

On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Igniters,
>
> I've added description of new plugin for Direct I/O for native
> persistence (
> https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> underthehood-DirectI/O
>
>
> SIncerely,
> Dmitriy Pavlov
>

[dmagda]

*Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
memory tuning page and refer to the wiki for more details:
https://issues.apache.org/jira/browse/IGNITE-7466

Please clarify the following:

> Direct I/O mode can't be enabled for Write Ahead Log files. However, when
> working with plugin, WAL manager applies advising Linux systems do not
> store the data of the file in page cache as they are not required.


For me, it means that WAL always goes through the operating system I/O
calls. Nothing changes for the WAL. However, I'm not sure what you meant to
explain by saying "when working with the plugin (Direct I/O) WAL manager
applies...". Could you rephrase it to bring more clarity?

*Raymond,*

If Direct I/O is enabled by default it will bring down the performance of
read-intensive application because, as Dmitry says, the reads bypass page
cache. So, I would recommend using it for write-intensive workloads and,
probably, for mixed-workloads depending on the reads and writes rate.

--
Denis


On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <[hidden email]>
wrote:

> Looks good!
>
> Is there any reason why this should not be a default setting if it
> gracefully downgrades to non-Direct IO if not supported by the OS?
>
> Thanks,
> Raymond.
>
> -----Original Message-----
> From: Dmitriy Setrakyan [mailto:[hidden email]]
> Sent: Wednesday, March 21, 2018 10:23 AM
> To: dev <[hidden email]>
> Subject: Re: Ignite Direct I/O plugin description added to wiki
>
> Thanks Dmitry, awesome work!
>
> On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Hi Igniters,
> >
> > I've added description of new plugin for Direct I/O for native
> > persistence (
> > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> > underthehood-DirectI/O
> >
> >
> > SIncerely,
> > Dmitriy Pavlov
> >
>

[Raymond Wilson]

Denis,

Thanks for the clarification on read-intensive workloads not being suitable
for Direct IO.

As a counterpoint to that, I would have thought the working set represented
by the in-memory data held by the Ignite node would be significantly larger
than the OS page cache which should mean the read hit rate on the OS page
cache would be very low under those workloads? Of course, this assumes your
WAL page size is configured to be similar to the OS page size, but still....

-----Original Message-----
From: Denis Magda [mailto:[hidden email]]
Sent: Wednesday, March 21, 2018 11:59 AM
To: [hidden email]; Prachi Garg <[hidden email]>
Subject: Re: Ignite Direct I/O plugin description added to wiki

*Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
memory tuning page and refer to the wiki for more details:
https://issues.apache.org/jira/browse/IGNITE-7466

Please clarify the following:

> Direct I/O mode can't be enabled for Write Ahead Log files. However,
> when working with plugin, WAL manager applies advising Linux systems
> do not store the data of the file in page cache as they are not required.


For me, it means that WAL always goes through the operating system I/O
calls. Nothing changes for the WAL. However, I'm not sure what you meant to
explain by saying "when working with the plugin (Direct I/O) WAL manager
applies...". Could you rephrase it to bring more clarity?

*Raymond,*

If Direct I/O is enabled by default it will bring down the performance of
read-intensive application because, as Dmitry says, the reads bypass page
cache. So, I would recommend using it for write-intensive workloads and,
probably, for mixed-workloads depending on the reads and writes rate.

--
Denis


On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <[hidden email]>
wrote:

> Looks good!
>
> Is there any reason why this should not be a default setting if it
> gracefully downgrades to non-Direct IO if not supported by the OS?
>
> Thanks,
> Raymond.
>
> -----Original Message-----
> From: Dmitriy Setrakyan [mailto:[hidden email]]
> Sent: Wednesday, March 21, 2018 10:23 AM
> To: dev <[hidden email]>
> Subject: Re: Ignite Direct I/O plugin description added to wiki
>
> Thanks Dmitry, awesome work!
>
> On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov
> <[hidden email]>
> wrote:
>
> > Hi Igniters,
> >
> > I've added description of new plugin for Direct I/O for native
> > persistence (
> > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> > underthehood-DirectI/O
> >
> >
> > SIncerely,
> > Dmitriy Pavlov
> >
>

[Dmitriy Pavlov]

Hi Igniters,

I've updated page with more detailed info about WAL and NativeIO, added
references to system properties.

Denis, thank you for noticing unclear section.

Sincerely,
Dmitriy Pavlov

ср, 21 мар. 2018 г. в 2:18, Raymond Wilson <[hidden email]>:

> Denis,
>
> Thanks for the clarification on read-intensive workloads not being suitable
> for Direct IO.
>
> As a counterpoint to that, I would have thought the working set represented
> by the in-memory data held by the Ignite node would be significantly larger
> than the OS page cache which should mean the read hit rate on the OS page
> cache would be very low under those workloads? Of course, this assumes your
> WAL page size is configured to be similar to the OS page size, but
> still....
>
> -----Original Message-----
> From: Denis Magda [mailto:[hidden email]]
> Sent: Wednesday, March 21, 2018 11:59 AM
> To: [hidden email]; Prachi Garg <[hidden email]>
> Subject: Re: Ignite Direct I/O plugin description added to wiki
>
> *Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
> memory tuning page and refer to the wiki for more details:
> https://issues.apache.org/jira/browse/IGNITE-7466
>
> Please clarify the following:
>
> > Direct I/O mode can't be enabled for Write Ahead Log files. However,
> > when working with plugin, WAL manager applies advising Linux systems
> > do not store the data of the file in page cache as they are not required.
>
>
> For me, it means that WAL always goes through the operating system I/O
> calls. Nothing changes for the WAL. However, I'm not sure what you meant to
> explain by saying "when working with the plugin (Direct I/O) WAL manager
> applies...". Could you rephrase it to bring more clarity?
>
> *Raymond,*
>
> If Direct I/O is enabled by default it will bring down the performance of
> read-intensive application because, as Dmitry says, the reads bypass page
> cache. So, I would recommend using it for write-intensive workloads and,
> probably, for mixed-workloads depending on the reads and writes rate.
>
> --
> Denis
>
>
> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
> [hidden email]>
> wrote:
>
> > Looks good!
> >
> > Is there any reason why this should not be a default setting if it
> > gracefully downgrades to non-Direct IO if not supported by the OS?
> >
> > Thanks,
> > Raymond.
> >
> > -----Original Message-----
> > From: Dmitriy Setrakyan [mailto:[hidden email]]
> > Sent: Wednesday, March 21, 2018 10:23 AM
> > To: dev <[hidden email]>
> > Subject: Re: Ignite Direct I/O plugin description added to wiki
> >
> > Thanks Dmitry, awesome work!
> >
> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov
> > <[hidden email]>
> > wrote:
> >
> > > Hi Igniters,
> > >
> > > I've added description of new plugin for Direct I/O for native
> > > persistence (
> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> > > underthehood-DirectI/O
> > >
> > >
> > > SIncerely,
> > > Dmitriy Pavlov
> > >
> >
>

[Dmitriy Pavlov]

Denis,

one more thing, can/should we mention Ignite Direct IO plugin in list of
project using JNA here: https://github.com/java-native-access/jna

Sincerely,
Dmitriy Pavlov

ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:

> *Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
> memory tuning page and refer to the wiki for more details:
> https://issues.apache.org/jira/browse/IGNITE-7466
>
> Please clarify the following:
>
> > Direct I/O mode can't be enabled for Write Ahead Log files. However, when
> > working with plugin, WAL manager applies advising Linux systems do not
> > store the data of the file in page cache as they are not required.
>
>
> For me, it means that WAL always goes through the operating system I/O
> calls. Nothing changes for the WAL. However, I'm not sure what you meant to
> explain by saying "when working with the plugin (Direct I/O) WAL manager
> applies...". Could you rephrase it to bring more clarity?
>
> *Raymond,*
>
> If Direct I/O is enabled by default it will bring down the performance of
> read-intensive application because, as Dmitry says, the reads bypass page
> cache. So, I would recommend using it for write-intensive workloads and,
> probably, for mixed-workloads depending on the reads and writes rate.
>
> --
> Denis
>
>
> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
> [hidden email]>
> wrote:
>
> > Looks good!
> >
> > Is there any reason why this should not be a default setting if it
> > gracefully downgrades to non-Direct IO if not supported by the OS?
> >
> > Thanks,
> > Raymond.
> >
> > -----Original Message-----
> > From: Dmitriy Setrakyan [mailto:[hidden email]]
> > Sent: Wednesday, March 21, 2018 10:23 AM
> > To: dev <[hidden email]>
> > Subject: Re: Ignite Direct I/O plugin description added to wiki
> >
> > Thanks Dmitry, awesome work!
> >
> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Hi Igniters,
> > >
> > > I've added description of new plugin for Direct I/O for native
> > > persistence (
> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> > > underthehood-DirectI/O
> > >
> > >
> > > SIncerely,
> > > Dmitriy Pavlov
> > >
> >
>

[dmagda]

Dmitriy,

Thanks for updating the WAL section. Now it makes total sense to me.

As for the page with JNA projects, let's get added there! Could you contact
the owners?

--
Denis


On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <[hidden email]>
wrote:

> Denis,
>
> one more thing, can/should we mention Ignite Direct IO plugin in list of
> project using JNA here: https://github.com/java-native-access/jna
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
>
>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
>> memory tuning page and refer to the wiki for more details:
>> https://issues.apache.org/jira/browse/IGNITE-7466
>>
>> Please clarify the following:
>>
>> > Direct I/O mode can't be enabled for Write Ahead Log files. However,
>> when
>> > working with plugin, WAL manager applies advising Linux systems do not
>> > store the data of the file in page cache as they are not required.
>>
>>
>> For me, it means that WAL always goes through the operating system I/O
>> calls. Nothing changes for the WAL. However, I'm not sure what you meant
>> to
>> explain by saying "when working with the plugin (Direct I/O) WAL manager
>> applies...". Could you rephrase it to bring more clarity?
>>
>> *Raymond,*
>>
>>
>> If Direct I/O is enabled by default it will bring down the performance of
>> read-intensive application because, as Dmitry says, the reads bypass page
>> cache. So, I would recommend using it for write-intensive workloads and,
>> probably, for mixed-workloads depending on the reads and writes rate.
>>
>> --
>> Denis
>>
>>
>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
>> [hidden email]>
>> wrote:
>>
>> > Looks good!
>> >
>> > Is there any reason why this should not be a default setting if it
>> > gracefully downgrades to non-Direct IO if not supported by the OS?
>> >
>> > Thanks,
>> > Raymond.
>> >
>> > -----Original Message-----
>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
>> > Sent: Wednesday, March 21, 2018 10:23 AM
>> > To: dev <[hidden email]>
>> > Subject: Re: Ignite Direct I/O plugin description added to wiki
>> >
>> > Thanks Dmitry, awesome work!
>> >
>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <[hidden email]>
>> > wrote:
>> >
>> > > Hi Igniters,
>> > >
>> > > I've added description of new plugin for Direct I/O for native
>> > > persistence (
>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
>> > > https://cwiki.apache.org/confluence/display/IGNITE/
>> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
>> > > underthehood-DirectI/O
>> > >
>> > >
>> > > SIncerely,
>> > > Dmitriy Pavlov
>> > >
>> >
>>
>

[Dmitriy Pavlov]

Yes, will do.

чт, 22 мар. 2018 г. в 1:05, Denis Magda <[hidden email]>:

> Dmitriy,
>
> Thanks for updating the WAL section. Now it makes total sense to me.
>
> As for the page with JNA projects, let's get added there! Could you
> contact the owners?
>
> --
> Denis
>
>
> On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <[hidden email]>
> wrote:
>
>> Denis,
>>
>> one more thing, can/should we mention Ignite Direct IO plugin in list of
>> project using JNA here: https://github.com/java-native-access/jna
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
>>
>>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
>>> memory tuning page and refer to the wiki for more details:
>>> https://issues.apache.org/jira/browse/IGNITE-7466
>>>
>>> Please clarify the following:
>>>
>>> > Direct I/O mode can't be enabled for Write Ahead Log files. However,
>>> when
>>> > working with plugin, WAL manager applies advising Linux systems do not
>>> > store the data of the file in page cache as they are not required.
>>>
>>>
>>> For me, it means that WAL always goes through the operating system I/O
>>> calls. Nothing changes for the WAL. However, I'm not sure what you meant
>>> to
>>> explain by saying "when working with the plugin (Direct I/O) WAL manager
>>> applies...". Could you rephrase it to bring more clarity?
>>>
>>> *Raymond,*
>>>
>>>
>>> If Direct I/O is enabled by default it will bring down the performance of
>>> read-intensive application because, as Dmitry says, the reads bypass page
>>> cache. So, I would recommend using it for write-intensive workloads and,
>>> probably, for mixed-workloads depending on the reads and writes rate.
>>>
>>> --
>>> Denis
>>>
>>>
>>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
>>> [hidden email]>
>>> wrote:
>>>
>>> > Looks good!
>>> >
>>> > Is there any reason why this should not be a default setting if it
>>> > gracefully downgrades to non-Direct IO if not supported by the OS?
>>> >
>>> > Thanks,
>>> > Raymond.
>>> >
>>> > -----Original Message-----
>>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
>>> > Sent: Wednesday, March 21, 2018 10:23 AM
>>> > To: dev <[hidden email]>
>>> > Subject: Re: Ignite Direct I/O plugin description added to wiki
>>> >
>>> > Thanks Dmitry, awesome work!
>>> >
>>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <[hidden email]
>>> >
>>> > wrote:
>>> >
>>> > > Hi Igniters,
>>> > >
>>> > > I've added description of new plugin for Direct I/O for native
>>> > > persistence (
>>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
>>> > > https://cwiki.apache.org/confluence/display/IGNITE/
>>> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
>>> > > underthehood-DirectI/O
>>> > >
>>> > >
>>> > > SIncerely,
>>> > > Dmitriy Pavlov
>>> > >
>>> >
>>>
>>
>

[Prachi Garg]

Dmitriy,

I have documented the Direct I/O plugin. Please review [1] and provide
comments/feedback in the ticket [2].

[1]
https://apacheignite.readme.io/v2.4/docs/durable-memory-tuning#section-enabling-direct-i-o

[2] https://issues.apache.org/jira/browse/IGNITE-7466


-Prachi

On Wed, Mar 21, 2018 at 11:22 PM, Dmitry Pavlov <[hidden email]>
wrote:

> Yes, will do.
>
> чт, 22 мар. 2018 г. в 1:05, Denis Magda <[hidden email]>:
>
> > Dmitriy,
> >
> > Thanks for updating the WAL section. Now it makes total sense to me.
> >
> > As for the page with JNA projects, let's get added there! Could you
> > contact the owners?
> >
> > --
> > Denis
> >
> >
> > On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> >> Denis,
> >>
> >> one more thing, can/should we mention Ignite Direct IO plugin in list of
> >> project using JNA here: https://github.com/java-native-access/jna
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
> >>
> >>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the durable
> >>> memory tuning page and refer to the wiki for more details:
> >>> https://issues.apache.org/jira/browse/IGNITE-7466
> >>>
> >>> Please clarify the following:
> >>>
> >>> > Direct I/O mode can't be enabled for Write Ahead Log files. However,
> >>> when
> >>> > working with plugin, WAL manager applies advising Linux systems do
> not
> >>> > store the data of the file in page cache as they are not required.
> >>>
> >>>
> >>> For me, it means that WAL always goes through the operating system I/O
> >>> calls. Nothing changes for the WAL. However, I'm not sure what you
> meant
> >>> to
> >>> explain by saying "when working with the plugin (Direct I/O) WAL
> manager
> >>> applies...". Could you rephrase it to bring more clarity?
> >>>
> >>> *Raymond,*
> >>>
> >>>
> >>> If Direct I/O is enabled by default it will bring down the performance
> of
> >>> read-intensive application because, as Dmitry says, the reads bypass
> page
> >>> cache. So, I would recommend using it for write-intensive workloads
> and,
> >>> probably, for mixed-workloads depending on the reads and writes rate.
> >>>
> >>> --
> >>> Denis
> >>>
> >>>
> >>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
> >>> [hidden email]>
> >>> wrote:
> >>>
> >>> > Looks good!
> >>> >
> >>> > Is there any reason why this should not be a default setting if it
> >>> > gracefully downgrades to non-Direct IO if not supported by the OS?
> >>> >
> >>> > Thanks,
> >>> > Raymond.
> >>> >
> >>> > -----Original Message-----
> >>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
> >>> > Sent: Wednesday, March 21, 2018 10:23 AM
> >>> > To: dev <[hidden email]>
> >>> > Subject: Re: Ignite Direct I/O plugin description added to wiki
> >>> >
> >>> > Thanks Dmitry, awesome work!
> >>> >
> >>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <
> [hidden email]
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Hi Igniters,
> >>> > >
> >>> > > I've added description of new plugin for Direct I/O for native
> >>> > > persistence (
> >>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> >>> > > https://cwiki.apache.org/confluence/display/IGNITE/
> >>> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> >>> > > underthehood-DirectI/O
> >>> > >
> >>> > >
> >>> > > SIncerely,
> >>> > > Dmitriy Pavlov
> >>> > >
> >>> >
> >>>
> >>
> >
>

[Dmitriy Pavlov]

Hi Prachi,

Yes, it's a great description.

I'm impressed by how informative and short you managed to make it.

Thank you!

Sincerely,
Dmitriy Pavlov

сб, 24 мар. 2018 г. в 0:29, Prachi Garg <[hidden email]>:

> Dmitriy,
>
> I have documented the Direct I/O plugin. Please review [1] and provide
> comments/feedback in the ticket [2].
>
> [1]
>
> https://apacheignite.readme.io/v2.4/docs/durable-memory-tuning#section-enabling-direct-i-o
>
> [2] https://issues.apache.org/jira/browse/IGNITE-7466
>
>
> -Prachi
>
> On Wed, Mar 21, 2018 at 11:22 PM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Yes, will do.
> >
> > чт, 22 мар. 2018 г. в 1:05, Denis Magda <[hidden email]>:
> >
> > > Dmitriy,
> > >
> > > Thanks for updating the WAL section. Now it makes total sense to me.
> > >
> > > As for the page with JNA projects, let's get added there! Could you
> > > contact the owners?
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > >
> > >> Denis,
> > >>
> > >> one more thing, can/should we mention Ignite Direct IO plugin in list
> of
> > >> project using JNA here: https://github.com/java-native-access/jna
> > >>
> > >> Sincerely,
> > >> Dmitriy Pavlov
> > >>
> > >> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
> > >>
> > >>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the
> durable
> > >>> memory tuning page and refer to the wiki for more details:
> > >>> https://issues.apache.org/jira/browse/IGNITE-7466
> > >>>
> > >>> Please clarify the following:
> > >>>
> > >>> > Direct I/O mode can't be enabled for Write Ahead Log files.
> However,
> > >>> when
> > >>> > working with plugin, WAL manager applies advising Linux systems do
> > not
> > >>> > store the data of the file in page cache as they are not required.
> > >>>
> > >>>
> > >>> For me, it means that WAL always goes through the operating system
> I/O
> > >>> calls. Nothing changes for the WAL. However, I'm not sure what you
> > meant
> > >>> to
> > >>> explain by saying "when working with the plugin (Direct I/O) WAL
> > manager
> > >>> applies...". Could you rephrase it to bring more clarity?
> > >>>
> > >>> *Raymond,*
> > >>>
> > >>>
> > >>> If Direct I/O is enabled by default it will bring down the
> performance
> > of
> > >>> read-intensive application because, as Dmitry says, the reads bypass
> > page
> > >>> cache. So, I would recommend using it for write-intensive workloads
> > and,
> > >>> probably, for mixed-workloads depending on the reads and writes rate.
> > >>>
> > >>> --
> > >>> Denis
> > >>>
> > >>>
> > >>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
> > >>> [hidden email]>
> > >>> wrote:
> > >>>
> > >>> > Looks good!
> > >>> >
> > >>> > Is there any reason why this should not be a default setting if it
> > >>> > gracefully downgrades to non-Direct IO if not supported by the OS?
> > >>> >
> > >>> > Thanks,
> > >>> > Raymond.
> > >>> >
> > >>> > -----Original Message-----
> > >>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
> > >>> > Sent: Wednesday, March 21, 2018 10:23 AM
> > >>> > To: dev <[hidden email]>
> > >>> > Subject: Re: Ignite Direct I/O plugin description added to wiki
> > >>> >
> > >>> > Thanks Dmitry, awesome work!
> > >>> >
> > >>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <
> > [hidden email]
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Igniters,
> > >>> > >
> > >>> > > I've added description of new plugin for Direct I/O for native
> > >>> > > persistence (
> > >>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> > >>> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > >>> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> > >>> > > underthehood-DirectI/O
> > >>> > >
> > >>> > >
> > >>> > > SIncerely,
> > >>> > > Dmitriy Pavlov
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >
> >
>

[dmagda]

Dmitriy, Prachi,

Thanks to both of you for the development and documentation of the feature.
I've added a reference from readme to Direct I/O section on wiki so that
people can learn even more about the capability.

--
Denis

On Fri, Mar 23, 2018 at 2:34 PM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Prachi,
>
> Yes, it's a great description.
>
> I'm impressed by how informative and short you managed to make it.
>
> Thank you!
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 24 мар. 2018 г. в 0:29, Prachi Garg <[hidden email]>:
>
>> Dmitriy,
>>
>> I have documented the Direct I/O plugin. Please review [1] and provide
>> comments/feedback in the ticket [2].
>>
>> [1]
>> https://apacheignite.readme.io/v2.4/docs/durable-memory-
>> tuning#section-enabling-direct-i-o
>>
>> [2] https://issues.apache.org/jira/browse/IGNITE-7466
>>
>>
>> -Prachi
>>
>> On Wed, Mar 21, 2018 at 11:22 PM, Dmitry Pavlov <[hidden email]>
>> wrote:
>>
>> > Yes, will do.
>> >
>> > чт, 22 мар. 2018 г. в 1:05, Denis Magda <[hidden email]>:
>> >
>> > > Dmitriy,
>> > >
>> > > Thanks for updating the WAL section. Now it makes total sense to me.
>> > >
>> > > As for the page with JNA projects, let's get added there! Could you
>> > > contact the owners?
>> > >
>> > > --
>> > > Denis
>> > >
>> > >
>> > > On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <[hidden email]
>> >
>> > > wrote:
>> > >
>> > >> Denis,
>> > >>
>> > >> one more thing, can/should we mention Ignite Direct IO plugin in
>> list of
>> > >> project using JNA here: https://github.com/java-native-access/jna
>> > >>
>> > >> Sincerely,
>> > >> Dmitriy Pavlov
>> > >>
>> > >> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
>> > >>
>> > >>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the
>> durable
>> > >>> memory tuning page and refer to the wiki for more details:
>> > >>> https://issues.apache.org/jira/browse/IGNITE-7466
>> > >>>
>> > >>> Please clarify the following:
>> > >>>
>> > >>> > Direct I/O mode can't be enabled for Write Ahead Log files.
>> However,
>> > >>> when
>> > >>> > working with plugin, WAL manager applies advising Linux systems do
>> > not
>> > >>> > store the data of the file in page cache as they are not required.
>> > >>>
>> > >>>
>> > >>> For me, it means that WAL always goes through the operating system
>> I/O
>> > >>> calls. Nothing changes for the WAL. However, I'm not sure what you
>> > meant
>> > >>> to
>> > >>> explain by saying "when working with the plugin (Direct I/O) WAL
>> > manager
>> > >>> applies...". Could you rephrase it to bring more clarity?
>> > >>>
>> > >>> *Raymond,*
>> > >>>
>> > >>>
>> > >>> If Direct I/O is enabled by default it will bring down the
>> performance
>> > of
>> > >>> read-intensive application because, as Dmitry says, the reads bypass
>> > page
>> > >>> cache. So, I would recommend using it for write-intensive workloads
>> > and,
>> > >>> probably, for mixed-workloads depending on the reads and writes
>> rate.
>> > >>>
>> > >>> --
>> > >>> Denis
>> > >>>
>> > >>>
>> > >>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
>> > >>> [hidden email]>
>> > >>> wrote:
>> > >>>
>> > >>> > Looks good!
>> > >>> >
>> > >>> > Is there any reason why this should not be a default setting if it
>> > >>> > gracefully downgrades to non-Direct IO if not supported by the OS?
>> > >>> >
>> > >>> > Thanks,
>> > >>> > Raymond.
>> > >>> >
>> > >>> > -----Original Message-----
>> > >>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
>> > >>> > Sent: Wednesday, March 21, 2018 10:23 AM
>> > >>> > To: dev <[hidden email]>
>> > >>> > Subject: Re: Ignite Direct I/O plugin description added to wiki
>> > >>> >
>> > >>> > Thanks Dmitry, awesome work!
>> > >>> >
>> > >>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <
>> > [hidden email]
>> > >>> >
>> > >>> > wrote:
>> > >>> >
>> > >>> > > Hi Igniters,
>> > >>> > >
>> > >>> > > I've added description of new plugin for Direct I/O for native
>> > >>> > > persistence (
>> > >>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
>> > >>> > > https://cwiki.apache.org/confluence/display/IGNITE/
>> > >>> > > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
>> > >>> > > underthehood-DirectI/O
>> > >>> > >
>> > >>> > >
>> > >>> > > SIncerely,
>> > >>> > > Dmitriy Pavlov
>> > >>> > >
>> > >>> >
>> > >>>
>> > >>
>> > >
>> >
>>
>

[Dmitriy Pavlov]

Hi Igniters,

Apache Ignite Entry was added to the projects list by JNA developers.

Now Apache Ignite is mentioned in main Java Native Access Readme.MD (
https://github.com/java-native-access/jna ), and link to Ignite site is
also available there.

Sincerely,
Dmitriy Pavlov


сб, 24 мар. 2018 г. в 3:33, Denis Magda <[hidden email]>:

> Dmitriy, Prachi,
>
> Thanks to both of you for the development and documentation of the feature.
> I've added a reference from readme to Direct I/O section on wiki so that
> people can learn even more about the capability.
>
> --
> Denis
>
> On Fri, Mar 23, 2018 at 2:34 PM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Hi Prachi,
> >
> > Yes, it's a great description.
> >
> > I'm impressed by how informative and short you managed to make it.
> >
> > Thank you!
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 24 мар. 2018 г. в 0:29, Prachi Garg <[hidden email]>:
> >
> >> Dmitriy,
> >>
> >> I have documented the Direct I/O plugin. Please review [1] and provide
> >> comments/feedback in the ticket [2].
> >>
> >> [1]
> >> https://apacheignite.readme.io/v2.4/docs/durable-memory-
> >> tuning#section-enabling-direct-i-o
> >>
> >> [2] https://issues.apache.org/jira/browse/IGNITE-7466
> >>
> >>
> >> -Prachi
> >>
> >> On Wed, Mar 21, 2018 at 11:22 PM, Dmitry Pavlov <[hidden email]>
> >> wrote:
> >>
> >> > Yes, will do.
> >> >
> >> > чт, 22 мар. 2018 г. в 1:05, Denis Magda <[hidden email]>:
> >> >
> >> > > Dmitriy,
> >> > >
> >> > > Thanks for updating the WAL section. Now it makes total sense to me.
> >> > >
> >> > > As for the page with JNA projects, let's get added there! Could you
> >> > > contact the owners?
> >> > >
> >> > > --
> >> > > Denis
> >> > >
> >> > >
> >> > > On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <
> [hidden email]
> >> >
> >> > > wrote:
> >> > >
> >> > >> Denis,
> >> > >>
> >> > >> one more thing, can/should we mention Ignite Direct IO plugin in
> >> list of
> >> > >> project using JNA here: https://github.com/java-native-access/jna
> >> > >>
> >> > >> Sincerely,
> >> > >> Dmitriy Pavlov
> >> > >>
> >> > >> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
> >> > >>
> >> > >>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the
> >> durable
> >> > >>> memory tuning page and refer to the wiki for more details:
> >> > >>> https://issues.apache.org/jira/browse/IGNITE-7466
> >> > >>>
> >> > >>> Please clarify the following:
> >> > >>>
> >> > >>> > Direct I/O mode can't be enabled for Write Ahead Log files.
> >> However,
> >> > >>> when
> >> > >>> > working with plugin, WAL manager applies advising Linux systems
> do
> >> > not
> >> > >>> > store the data of the file in page cache as they are not
> required.
> >> > >>>
> >> > >>>
> >> > >>> For me, it means that WAL always goes through the operating system
> >> I/O
> >> > >>> calls. Nothing changes for the WAL. However, I'm not sure what you
> >> > meant
> >> > >>> to
> >> > >>> explain by saying "when working with the plugin (Direct I/O) WAL
> >> > manager
> >> > >>> applies...". Could you rephrase it to bring more clarity?
> >> > >>>
> >> > >>> *Raymond,*
> >> > >>>
> >> > >>>
> >> > >>> If Direct I/O is enabled by default it will bring down the
> >> performance
> >> > of
> >> > >>> read-intensive application because, as Dmitry says, the reads
> bypass
> >> > page
> >> > >>> cache. So, I would recommend using it for write-intensive
> workloads
> >> > and,
> >> > >>> probably, for mixed-workloads depending on the reads and writes
> >> rate.
> >> > >>>
> >> > >>> --
> >> > >>> Denis
> >> > >>>
> >> > >>>
> >> > >>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
> >> > >>> [hidden email]>
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Looks good!
> >> > >>> >
> >> > >>> > Is there any reason why this should not be a default setting if
> it
> >> > >>> > gracefully downgrades to non-Direct IO if not supported by the
> OS?
> >> > >>> >
> >> > >>> > Thanks,
> >> > >>> > Raymond.
> >> > >>> >
> >> > >>> > -----Original Message-----
> >> > >>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
> >> > >>> > Sent: Wednesday, March 21, 2018 10:23 AM
> >> > >>> > To: dev <[hidden email]>
> >> > >>> > Subject: Re: Ignite Direct I/O plugin description added to wiki
> >> > >>> >
> >> > >>> > Thanks Dmitry, awesome work!
> >> > >>> >
> >> > >>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <
> >> > [hidden email]
> >> > >>> >
> >> > >>> > wrote:
> >> > >>> >
> >> > >>> > > Hi Igniters,
> >> > >>> > >
> >> > >>> > > I've added description of new plugin for Direct I/O for native
> >> > >>> > > persistence (
> >> > >>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> >> > >>> > > https://cwiki.apache.org/confluence/display/IGNITE/
> >> > >>> > >
> Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> >> > >>> > > underthehood-DirectI/O
> >> > >>> > >
> >> > >>> > >
> >> > >>> > > SIncerely,
> >> > >>> > > Dmitriy Pavlov
> >> > >>> > >
> >> > >>> >
> >> > >>>
> >> > >>
> >> > >
> >> >
> >>
> >
>

[dsetrakyan]

Nice!

On Sun, Mar 25, 2018 at 2:57 PM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Igniters,
>
> Apache Ignite Entry was added to the projects list by JNA developers.
>
> Now Apache Ignite is mentioned in main Java Native Access Readme.MD (
> https://github.com/java-native-access/jna ), and link to Ignite site is
> also available there.
>
> Sincerely,
> Dmitriy Pavlov
>
>
> сб, 24 мар. 2018 г. в 3:33, Denis Magda <[hidden email]>:
>
> > Dmitriy, Prachi,
> >
> > Thanks to both of you for the development and documentation of the
> feature.
> > I've added a reference from readme to Direct I/O section on wiki so that
> > people can learn even more about the capability.
> >
> > --
> > Denis
> >
> > On Fri, Mar 23, 2018 at 2:34 PM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Hi Prachi,
> > >
> > > Yes, it's a great description.
> > >
> > > I'm impressed by how informative and short you managed to make it.
> > >
> > > Thank you!
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > сб, 24 мар. 2018 г. в 0:29, Prachi Garg <[hidden email]>:
> > >
> > >> Dmitriy,
> > >>
> > >> I have documented the Direct I/O plugin. Please review [1] and provide
> > >> comments/feedback in the ticket [2].
> > >>
> > >> [1]
> > >> https://apacheignite.readme.io/v2.4/docs/durable-memory-
> > >> tuning#section-enabling-direct-i-o
> > >>
> > >> [2] https://issues.apache.org/jira/browse/IGNITE-7466
> > >>
> > >>
> > >> -Prachi
> > >>
> > >> On Wed, Mar 21, 2018 at 11:22 PM, Dmitry Pavlov <
> [hidden email]>
> > >> wrote:
> > >>
> > >> > Yes, will do.
> > >> >
> > >> > чт, 22 мар. 2018 г. в 1:05, Denis Magda <[hidden email]>:
> > >> >
> > >> > > Dmitriy,
> > >> > >
> > >> > > Thanks for updating the WAL section. Now it makes total sense to
> me.
> > >> > >
> > >> > > As for the page with JNA projects, let's get added there! Could
> you
> > >> > > contact the owners?
> > >> > >
> > >> > > --
> > >> > > Denis
> > >> > >
> > >> > >
> > >> > > On Wed, Mar 21, 2018 at 9:27 AM, Dmitry Pavlov <
> > [hidden email]
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> Denis,
> > >> > >>
> > >> > >> one more thing, can/should we mention Ignite Direct IO plugin in
> > >> list of
> > >> > >> project using JNA here: https://github.com/java-
> native-access/jna
> > >> > >>
> > >> > >> Sincerely,
> > >> > >> Dmitriy Pavlov
> > >> > >>
> > >> > >> ср, 21 мар. 2018 г. в 1:59, Denis Magda <[hidden email]>:
> > >> > >>
> > >> > >>> *Dmitriy*, thanks. Astonishing job! We'll add a section to the
> > >> durable
> > >> > >>> memory tuning page and refer to the wiki for more details:
> > >> > >>> https://issues.apache.org/jira/browse/IGNITE-7466
> > >> > >>>
> > >> > >>> Please clarify the following:
> > >> > >>>
> > >> > >>> > Direct I/O mode can't be enabled for Write Ahead Log files.
> > >> However,
> > >> > >>> when
> > >> > >>> > working with plugin, WAL manager applies advising Linux
> systems
> > do
> > >> > not
> > >> > >>> > store the data of the file in page cache as they are not
> > required.
> > >> > >>>
> > >> > >>>
> > >> > >>> For me, it means that WAL always goes through the operating
> system
> > >> I/O
> > >> > >>> calls. Nothing changes for the WAL. However, I'm not sure what
> you
> > >> > meant
> > >> > >>> to
> > >> > >>> explain by saying "when working with the plugin (Direct I/O) WAL
> > >> > manager
> > >> > >>> applies...". Could you rephrase it to bring more clarity?
> > >> > >>>
> > >> > >>> *Raymond,*
> > >> > >>>
> > >> > >>>
> > >> > >>> If Direct I/O is enabled by default it will bring down the
> > >> performance
> > >> > of
> > >> > >>> read-intensive application because, as Dmitry says, the reads
> > bypass
> > >> > page
> > >> > >>> cache. So, I would recommend using it for write-intensive
> > workloads
> > >> > and,
> > >> > >>> probably, for mixed-workloads depending on the reads and writes
> > >> rate.
> > >> > >>>
> > >> > >>> --
> > >> > >>> Denis
> > >> > >>>
> > >> > >>>
> > >> > >>> On Tue, Mar 20, 2018 at 2:29 PM, Raymond Wilson <
> > >> > >>> [hidden email]>
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>> > Looks good!
> > >> > >>> >
> > >> > >>> > Is there any reason why this should not be a default setting
> if
> > it
> > >> > >>> > gracefully downgrades to non-Direct IO if not supported by the
> > OS?
> > >> > >>> >
> > >> > >>> > Thanks,
> > >> > >>> > Raymond.
> > >> > >>> >
> > >> > >>> > -----Original Message-----
> > >> > >>> > From: Dmitriy Setrakyan [mailto:[hidden email]]
> > >> > >>> > Sent: Wednesday, March 21, 2018 10:23 AM
> > >> > >>> > To: dev <[hidden email]>
> > >> > >>> > Subject: Re: Ignite Direct I/O plugin description added to
> wiki
> > >> > >>> >
> > >> > >>> > Thanks Dmitry, awesome work!
> > >> > >>> >
> > >> > >>> > On Wed, Mar 21, 2018 at 12:21 AM, Dmitry Pavlov <
> > >> > [hidden email]
> > >> > >>> >
> > >> > >>> > wrote:
> > >> > >>> >
> > >> > >>> > > Hi Igniters,
> > >> > >>> > >
> > >> > >>> > > I've added description of new plugin for Direct I/O for
> native
> > >> > >>> > > persistence (
> > >> > >>> > > https://issues.apache.org/jira/browse/IGNITE-6341)  to wiki
> > >> > >>> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > >> > >>> > >
> > Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-
> > >> > >>> > > underthehood-DirectI/O
> > >> > >>> > >
> > >> > >>> > >
> > >> > >>> > > SIncerely,
> > >> > >>> > > Dmitriy Pavlov
> > >> > >>> > >
> > >> > >>> >
> > >> > >>>
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
> >
>