Durable Memory and Ignite Persistence Tuning

[dmagda]

Igniters,

I see a lot of complains regarding the performance of the subj on the user list. At the same time, I do believe that in most scenarios it’s a lack of knowledge that we keep in secret.

It's time to document Durable Memory and its Native Persistence tuning parameters. Let's start doing this for Linux based deployments first. Here is what we have for now (which is almost nothing): https://apacheignite.readme.io/docs/durable-memory-tuning <https://apacheignite.readme.io/docs/durable-memory-tuning>

Ideally, at some point we have to come up with doc like this: https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf

Please share your expertise in a form of settings that have to be put on the paper. We put them in JIRA and document afterwords:
https://issues.apache.org/jira/browse/IGNITE-6246 <https://issues.apache.org/jira/browse/IGNITE-6246>

—
Denis

[Konstantin Boudnik-2]

Just to spice it up: in my experience, having a few hundred parameters one can
tweak (I am making up the number here) is a tough UX call. In JVM, we had a
team that worked on heuristics that would auto-tune a bunch of things during
the VM startup. Hence, providing the best user experience in most cases.

Do you think something like this is feasible in case of persistence
functionality? Documenting it first is a great first step, but perhaps
wrapping this knowledge into some soft of code, would be even better.

I do have an anecdotal evidence where an experienced Ignite developer
tried to implement a message processing system with Ignite 2.0. After a week
of getting nowhere performance wise, he dropped it and implemented something
custom with Java and nothing else. Take it or leave it: that's why I called
this "anecdotal" in the first place.

Speaking strictly for myself: when I come across blog posts about tuning of
Apache Kudu or Apache Impala the skin on the back of head starts crawling. I
imagine 95% of the potential users of these applications would turn away right
at that point. If the system doesn't work well enough out of the box - screw
it, there's 355 other alternatives available in the FOSS market.

Thoughts?
  Cos

On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:

> Igniters,
>
> I see a lot of complains regarding the performance of the subj on the user
> list. At the same time, I do believe that in most scenarios it’s a lack of
> knowledge that we keep in secret.
>
> It's time to document Durable Memory and its Native Persistence tuning
> parameters. Let's start doing this for Linux based deployments first. Here
> is what we have for now (which is almost nothing):
> https://apacheignite.readme.io/docs/durable-memory-tuning
> <https://apacheignite.readme.io/docs/durable-memory-tuning>
>
> Ideally, at some point we have to come up with doc like this:
> https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>
> Please share your expertise in a form of settings that have to be put on the
> paper. We put them in JIRA and document afterwords:
> https://issues.apache.org/jira/browse/IGNITE-6246
> <https://issues.apache.org/jira/browse/IGNITE-6246>
>
> —
> Denis

[dmagda]

Cos,

Sure, as I see from the @dev list a lot of efforts are put to the auto-tune way. We need to keep to this way whenever it’s possible.

However, it might be the case that we can’t auto-tune some parameters (Linux kernel behavior) or we know that setting A will be auto-tuned in release 2.x but for now we should explain how to set it manually.

Even more, we can review that page over the time and decide that now we’re ready to auto-tune parameter B.

—
Denis

> On Sep 1, 2017, at 12:02 PM, Konstantin Boudnik <[hidden email]> wrote:
>
> Just to spice it up: in my experience, having a few hundred parameters one can
> tweak (I am making up the number here) is a tough UX call. In JVM, we had a
> team that worked on heuristics that would auto-tune a bunch of things during
> the VM startup. Hence, providing the best user experience in most cases.
>
> Do you think something like this is feasible in case of persistence
> functionality? Documenting it first is a great first step, but perhaps
> wrapping this knowledge into some soft of code, would be even better.
>
> I do have an anecdotal evidence where an experienced Ignite developer
> tried to implement a message processing system with Ignite 2.0. After a week
> of getting nowhere performance wise, he dropped it and implemented something
> custom with Java and nothing else. Take it or leave it: that's why I called
> this "anecdotal" in the first place.
>
> Speaking strictly for myself: when I come across blog posts about tuning of
> Apache Kudu or Apache Impala the skin on the back of head starts crawling. I
> imagine 95% of the potential users of these applications would turn away right
> at that point. If the system doesn't work well enough out of the box - screw
> it, there's 355 other alternatives available in the FOSS market.
>
> Thoughts?
>  Cos
>
> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the user
>> list. At the same time, I do believe that in most scenarios it’s a lack of
>> knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence tuning
>> parameters. Let's start doing this for Linux based deployments first. Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning
>> <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>
>> Ideally, at some point we have to come up with doc like this:
>> https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put on the
>> paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246
>> <https://issues.apache.org/jira/browse/IGNITE-6246>
>>
>> —
>> Denis

[nivanov]

200% agree.

UX is major problem for Ignite (especially so for 2.0 with all major
redev). Removing (!) configuration properties should be THE GOAL, not
adding new one or documenting them (which is a crutch to begin with).

When I see a product with dozens config properties or more I desperately
look for ways not to use it...

My 2 cents,
--
Nikita Ivanov


On Fri, Sep 1, 2017 at 12:02 PM, Konstantin Boudnik <[hidden email]> wrote:

> Just to spice it up: in my experience, having a few hundred parameters one
> can
> tweak (I am making up the number here) is a tough UX call. In JVM, we had a
> team that worked on heuristics that would auto-tune a bunch of things
> during
> the VM startup. Hence, providing the best user experience in most cases.
>
> Do you think something like this is feasible in case of persistence
> functionality? Documenting it first is a great first step, but perhaps
> wrapping this knowledge into some soft of code, would be even better.
>
> I do have an anecdotal evidence where an experienced Ignite developer
> tried to implement a message processing system with Ignite 2.0. After a
> week
> of getting nowhere performance wise, he dropped it and implemented
> something
> custom with Java and nothing else. Take it or leave it: that's why I called
> this "anecdotal" in the first place.
>
> Speaking strictly for myself: when I come across blog posts about tuning of
> Apache Kudu or Apache Impala the skin on the back of head starts crawling.
> I
> imagine 95% of the potential users of these applications would turn away
> right
> at that point. If the system doesn't work well enough out of the box -
> screw
> it, there's 355 other alternatives available in the FOSS market.
>
> Thoughts?
>   Cos
>
> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
> > Igniters,
> >
> > I see a lot of complains regarding the performance of the subj on the
> user
> > list. At the same time, I do believe that in most scenarios it’s a lack
> of
> > knowledge that we keep in secret.
> >
> > It's time to document Durable Memory and its Native Persistence tuning
> > parameters. Let's start doing this for Linux based deployments first.
> Here
> > is what we have for now (which is almost nothing):
> > https://apacheignite.readme.io/docs/durable-memory-tuning
> > <https://apacheignite.readme.io/docs/durable-memory-tuning>
> >
> > Ideally, at some point we have to come up with doc like this:
> > https://access.redhat.com/sites/default/files/
> attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
> >
> > Please share your expertise in a form of settings that have to be put on
> the
> > paper. We put them in JIRA and document afterwords:
> > https://issues.apache.org/jira/browse/IGNITE-6246
> > <https://issues.apache.org/jira/browse/IGNITE-6246>
> >
> > —
> > Denis
>

[dmagda]

It doesn’t matter how many configuration parameters your platform, database or operating system has - the performance tuning usually takes place in business deployments. The performance optimizations might happen behind the scene by heuristic algorithms or basic checks or be explained in performance guides or both.

This discussion is about this exact guide that is vital for DevOps and Admins. Make it first and you’ll reveal the spots for auto-tuning. Otherwise, we can spend months keeping a blind eye on this problem waiting for fully automated heaven and cleanest API in the world but nothing will happen at all.

—
Denis  

> On Sep 1, 2017, at 12:49 PM, Nikita Ivanov <[hidden email]> wrote:
>
> 200% agree.
>
> UX is major problem for Ignite (especially so for 2.0 with all major
> redev). Removing (!) configuration properties should be THE GOAL, not
> adding new one or documenting them (which is a crutch to begin with).
>
> When I see a product with dozens config properties or more I desperately
> look for ways not to use it...
>
> My 2 cents,
> --
> Nikita Ivanov
>
>
> On Fri, Sep 1, 2017 at 12:02 PM, Konstantin Boudnik <[hidden email]> wrote:
>
>> Just to spice it up: in my experience, having a few hundred parameters one
>> can
>> tweak (I am making up the number here) is a tough UX call. In JVM, we had a
>> team that worked on heuristics that would auto-tune a bunch of things
>> during
>> the VM startup. Hence, providing the best user experience in most cases.
>>
>> Do you think something like this is feasible in case of persistence
>> functionality? Documenting it first is a great first step, but perhaps
>> wrapping this knowledge into some soft of code, would be even better.
>>
>> I do have an anecdotal evidence where an experienced Ignite developer
>> tried to implement a message processing system with Ignite 2.0. After a
>> week
>> of getting nowhere performance wise, he dropped it and implemented
>> something
>> custom with Java and nothing else. Take it or leave it: that's why I called
>> this "anecdotal" in the first place.
>>
>> Speaking strictly for myself: when I come across blog posts about tuning of
>> Apache Kudu or Apache Impala the skin on the back of head starts crawling.
>> I
>> imagine 95% of the potential users of these applications would turn away
>> right
>> at that point. If the system doesn't work well enough out of the box -
>> screw
>> it, there's 355 other alternatives available in the FOSS market.
>>
>> Thoughts?
>>  Cos
>>
>> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>>> Igniters,
>>>
>>> I see a lot of complains regarding the performance of the subj on the
>> user
>>> list. At the same time, I do believe that in most scenarios it’s a lack
>> of
>>> knowledge that we keep in secret.
>>>
>>> It's time to document Durable Memory and its Native Persistence tuning
>>> parameters. Let's start doing this for Linux based deployments first.
>> Here
>>> is what we have for now (which is almost nothing):
>>> https://apacheignite.readme.io/docs/durable-memory-tuning
>>> <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>>
>>> Ideally, at some point we have to come up with doc like this:
>>> https://access.redhat.com/sites/default/files/
>> attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>>
>>> Please share your expertise in a form of settings that have to be put on
>> the
>>> paper. We put them in JIRA and document afterwords:
>>> https://issues.apache.org/jira/browse/IGNITE-6246
>>> <https://issues.apache.org/jira/browse/IGNITE-6246>
>>>
>>> —
>>> Denis
>>

[nivanov]

My points was in support of auto-tune for Ignite's internals. As far as
environment, sure, we need to have a doc.
--
Nikita Ivanov


On Fri, Sep 1, 2017 at 1:23 PM, Denis Magda <[hidden email]> wrote:

> It doesn’t matter how many configuration parameters your platform,
> database or operating system has - the performance tuning usually takes
> place in business deployments. The performance optimizations might happen
> behind the scene by heuristic algorithms or basic checks or be explained in
> performance guides or both.
>
> This discussion is about this exact guide that is vital for DevOps and
> Admins. Make it first and you’ll reveal the spots for auto-tuning.
> Otherwise, we can spend months keeping a blind eye on this problem waiting
> for fully automated heaven and cleanest API in the world but nothing will
> happen at all.
>
> —
> Denis
>
> > On Sep 1, 2017, at 12:49 PM, Nikita Ivanov <[hidden email]> wrote:
> >
> > 200% agree.
> >
> > UX is major problem for Ignite (especially so for 2.0 with all major
> > redev). Removing (!) configuration properties should be THE GOAL, not
> > adding new one or documenting them (which is a crutch to begin with).
> >
> > When I see a product with dozens config properties or more I desperately
> > look for ways not to use it...
> >
> > My 2 cents,
> > --
> > Nikita Ivanov
> >
> >
> > On Fri, Sep 1, 2017 at 12:02 PM, Konstantin Boudnik <[hidden email]>
> wrote:
> >
> >> Just to spice it up: in my experience, having a few hundred parameters
> one
> >> can
> >> tweak (I am making up the number here) is a tough UX call. In JVM, we
> had a
> >> team that worked on heuristics that would auto-tune a bunch of things
> >> during
> >> the VM startup. Hence, providing the best user experience in most cases.
> >>
> >> Do you think something like this is feasible in case of persistence
> >> functionality? Documenting it first is a great first step, but perhaps
> >> wrapping this knowledge into some soft of code, would be even better.
> >>
> >> I do have an anecdotal evidence where an experienced Ignite developer
> >> tried to implement a message processing system with Ignite 2.0. After a
> >> week
> >> of getting nowhere performance wise, he dropped it and implemented
> >> something
> >> custom with Java and nothing else. Take it or leave it: that's why I
> called
> >> this "anecdotal" in the first place.
> >>
> >> Speaking strictly for myself: when I come across blog posts about
> tuning of
> >> Apache Kudu or Apache Impala the skin on the back of head starts
> crawling.
> >> I
> >> imagine 95% of the potential users of these applications would turn away
> >> right
> >> at that point. If the system doesn't work well enough out of the box -
> >> screw
> >> it, there's 355 other alternatives available in the FOSS market.
> >>
> >> Thoughts?
> >>  Cos
> >>
> >> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
> >>> Igniters,
> >>>
> >>> I see a lot of complains regarding the performance of the subj on the
> >> user
> >>> list. At the same time, I do believe that in most scenarios it’s a lack
> >> of
> >>> knowledge that we keep in secret.
> >>>
> >>> It's time to document Durable Memory and its Native Persistence tuning
> >>> parameters. Let's start doing this for Linux based deployments first.
> >> Here
> >>> is what we have for now (which is almost nothing):
> >>> https://apacheignite.readme.io/docs/durable-memory-tuning
> >>> <https://apacheignite.readme.io/docs/durable-memory-tuning>
> >>>
> >>> Ideally, at some point we have to come up with doc like this:
> >>> https://access.redhat.com/sites/default/files/
> >> attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
> >>>
> >>> Please share your expertise in a form of settings that have to be put
> on
> >> the
> >>> paper. We put them in JIRA and document afterwords:
> >>> https://issues.apache.org/jira/browse/IGNITE-6246
> >>> <https://issues.apache.org/jira/browse/IGNITE-6246>
> >>>
> >>> —
> >>> Denis
> >>
>
>

[dsetrakyan]

Cos, great point!

I think the Ignite community should start load testing with default settings, without any config changes. This should open a lot of holes.

⁣D.​

On Sep 1, 2017, 9:57 PM, at 9:57 PM, Konstantin Boudnik <[hidden email]> wrote:

>Just to spice it up: in my experience, having a few hundred parameters
>one can
>tweak (I am making up the number here) is a tough UX call. In JVM, we
>had a
>team that worked on heuristics that would auto-tune a bunch of things
>during
>the VM startup. Hence, providing the best user experience in most
>cases.
>
>Do you think something like this is feasible in case of persistence
>functionality? Documenting it first is a great first step, but perhaps
>wrapping this knowledge into some soft of code, would be even better.
>
>I do have an anecdotal evidence where an experienced Ignite developer
>tried to implement a message processing system with Ignite 2.0. After a
>week
>of getting nowhere performance wise, he dropped it and implemented
>something
>custom with Java and nothing else. Take it or leave it: that's why I
>called
>this "anecdotal" in the first place.
>
>Speaking strictly for myself: when I come across blog posts about
>tuning of
>Apache Kudu or Apache Impala the skin on the back of head starts
>crawling. I
>imagine 95% of the potential users of these applications would turn
>away right
>at that point. If the system doesn't work well enough out of the box -
>screw
>it, there's 355 other alternatives available in the FOSS market.
>
>Thoughts?
>  Cos
>
>On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the
>user
>> list. At the same time, I do believe that in most scenarios it’s a
>lack of
>> knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence
>tuning
>> parameters. Let's start doing this for Linux based deployments first.
>Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning
>> <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>
>> Ideally, at some point we have to come up with doc like this:
>>
>https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put
>on the
>> paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246
>> <https://issues.apache.org/jira/browse/IGNITE-6246>
>>
>> —
>> Denis

[Ivan Rakov]

Folks,

We had some experience of benchmarking Ignite with persistent store on
SSD. I think we can share some helpful advice. None of them require
changing configuration of Ignite or persistent store.

*Tuning advice for users*

1) Be prepared for LFS performance decrease after several hours of
intensive load. Unfortunately, that's how SSD drives work:
http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
Consider buying fast production-level SSD drives.
2) Consider using separate drives for LFS files and WAL. Ignite actively
performs writes to both LFS and WAL under intensive load, and having two
devices will double your throughput limit.
3) Over-provision your SSD. Performance of random writes on 50% filled
disk is much better than on 90% filled. SSD Over-Provisioning And Its
Benefits:
http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
4) Leave free space in RAM to let OS use page cache and optimize writes.
Total size of all memory policies shouldn't exceed 70% of your RAM.
5) Make sure that OS doesn't utilize swap. If you use Unix, best option
is set vm.swappiness to 0.
6) Try to find out page size of your SSD. Ideally, page size of Ignite
shouldn't be less than SSD page size. Possible approaches:
Find it in device specification (some manufacturers don't reveal it)
Try running SSD benchmarks
If you are not sure, just set page size to 4K. As various benchmarks use
4K pages, manufacturers have to adapt drives for 4K random write
workload. Whitepaper from Intel showing that 4K pages are enough:
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
Check your OS page cache size. Page size of Ignite shouldn't be less
than OS page size. How to check OS cache page size in Unix:
https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space


Best Regards,
Ivan Rakov

On 01.09.2017 21:08, Denis Magda wrote:

> Igniters,
>
> I see a lot of complains regarding the performance of the subj on the user list. At the same time, I do believe that in most scenarios it’s a lack of knowledge that we keep in secret.
>
> It's time to document Durable Memory and its Native Persistence tuning parameters. Let's start doing this for Linux based deployments first. Here is what we have for now (which is almost nothing): https://apacheignite.readme.io/docs/durable-memory-tuning <https://apacheignite.readme.io/docs/durable-memory-tuning>
>
> Ideally, at some point we have to come up with doc like this: https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>
> Please share your expertise in a form of settings that have to be put on the paper. We put them in JIRA and document afterwords:
> https://issues.apache.org/jira/browse/IGNITE-6246 <https://issues.apache.org/jira/browse/IGNITE-6246>
>
> —
> Denis

[Ivan Rakov]

 > None of them require changing configuration of Ignite or persistent store
Disclaimer: this will be true since 2.3.

https://issues.apache.org/jira/browse/IGNITE-6182 - memory policy
default max size set to 20% of RAM - since 2.2
https://issues.apache.org/jira/browse/IGNITE-5884 - default pageSize set
to 4K - since 2.3

Best Regards,
Ivan Rakov


On 13.09.2017 12:29, Ivan Rakov wrote:

> Folks,
>
> We had some experience of benchmarking Ignite with persistent store on
> SSD. I think we can share some helpful advice. None of them require
> changing configuration of Ignite or persistent store.
>
> *Tuning advice for users*
>
> 1) Be prepared for LFS performance decrease after several hours of
> intensive load. Unfortunately, that's how SSD drives work:
> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
> Consider buying fast production-level SSD drives.
> 2) Consider using separate drives for LFS files and WAL. Ignite
> actively performs writes to both LFS and WAL under intensive load, and
> having two devices will double your throughput limit.
> 3) Over-provision your SSD. Performance of random writes on 50% filled
> disk is much better than on 90% filled. SSD Over-Provisioning And Its
> Benefits:
> http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
> 4) Leave free space in RAM to let OS use page cache and optimize
> writes. Total size of all memory policies shouldn't exceed 70% of your
> RAM.
> 5) Make sure that OS doesn't utilize swap. If you use Unix, best
> option is set vm.swappiness to 0.
> 6) Try to find out page size of your SSD. Ideally, page size of Ignite
> shouldn't be less than SSD page size. Possible approaches:
> Find it in device specification (some manufacturers don't reveal it)
> Try running SSD benchmarks
> If you are not sure, just set page size to 4K. As various benchmarks
> use 4K pages, manufacturers have to adapt drives for 4K random write
> workload. Whitepaper from Intel showing that 4K pages are enough:
> https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
> Check your OS page cache size. Page size of Ignite shouldn't be less
> than OS page size. How to check OS cache page size in Unix:
> https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
>
>
> Best Regards,
> Ivan Rakov
>
> On 01.09.2017 21:08, Denis Magda wrote:
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the
>> user list. At the same time, I do believe that in most scenarios it’s
>> a lack of knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence
>> tuning parameters. Let's start doing this for Linux based deployments
>> first. Here is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning 
>> <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>
>> Ideally, at some point we have to come up with doc like this:
>> https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put
>> on the paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246 
>> <https://issues.apache.org/jira/browse/IGNITE-6246>
>>
>> —
>> Denis
>
>

[nivanov]

Excellent work on this... This should be expanded and be prominently placed
in our docs/tutorials/javadocs/etc.

--
Nikita Ivanov


On Wed, Sep 13, 2017 at 12:29 PM, Ivan Rakov <[hidden email]> wrote:

> Folks,
>
> We had some experience of benchmarking Ignite with persistent store on
> SSD. I think we can share some helpful advice. None of them require
> changing configuration of Ignite or persistent store.
>
> *Tuning advice for users*
>
> 1) Be prepared for LFS performance decrease after several hours of
> intensive load. Unfortunately, that's how SSD drives work:
> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-arc
> hitecture-of-an-ssd-and-benchmarking/
> Consider buying fast production-level SSD drives.
> 2) Consider using separate drives for LFS files and WAL. Ignite actively
> performs writes to both LFS and WAL under intensive load, and having two
> devices will double your throughput limit.
> 3) Over-provision your SSD. Performance of random writes on 50% filled
> disk is much better than on 90% filled. SSD Over-Provisioning And Its
> Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisio
> ning-benefits-master-ti/
> 4) Leave free space in RAM to let OS use page cache and optimize writes.
> Total size of all memory policies shouldn't exceed 70% of your RAM.
> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is
> set vm.swappiness to 0.
> 6) Try to find out page size of your SSD. Ideally, page size of Ignite
> shouldn't be less than SSD page size. Possible approaches:
> Find it in device specification (some manufacturers don't reveal it)
> Try running SSD benchmarks
> If you are not sure, just set page size to 4K. As various benchmarks use
> 4K pages, manufacturers have to adapt drives for 4K random write workload.
> Whitepaper from Intel showing that 4K pages are enough:
> https://www.intel.com/content/dam/www/public/us/en/documents
> /white-papers/ssd-server-storage-applications-paper.pdf
> Check your OS page cache size. Page size of Ignite shouldn't be less than
> OS page size. How to check OS cache page size in Unix:
> https://unix.stackexchange.com/questions/128213/how-is-page-
> size-determined-in-virtual-address-space
>
>
> Best Regards,
> Ivan Rakov
>
> On 01.09.2017 21:08, Denis Magda wrote:
>
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the
>> user list. At the same time, I do believe that in most scenarios it’s a
>> lack of knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence tuning
>> parameters. Let's start doing this for Linux based deployments first. Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning <
>> https://apacheignite.readme.io/docs/durable-memory-tuning>
>>
>> Ideally, at some point we have to come up with doc like this:
>> https://access.redhat.com/sites/default/files/attachments/
>> deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put on
>> the paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246 <
>> https://issues.apache.org/jira/browse/IGNITE-6246>
>>
>> —
>> Denis
>>
>
>

[dsetrakyan]

Completely agree with Nikita. Why not add this information here?

https://apacheignite.readme.io/docs/durable-memory-tuning

D.

On Wed, Sep 13, 2017 at 2:54 AM, Nikita Ivanov <[hidden email]> wrote:

> Excellent work on this... This should be expanded and be prominently placed
> in our docs/tutorials/javadocs/etc.
>
> --
> Nikita Ivanov
>
>
> On Wed, Sep 13, 2017 at 12:29 PM, Ivan Rakov <[hidden email]>
> wrote:
>
> > Folks,
> >
> > We had some experience of benchmarking Ignite with persistent store on
> > SSD. I think we can share some helpful advice. None of them require
> > changing configuration of Ignite or persistent store.
> >
> > *Tuning advice for users*
> >
> > 1) Be prepared for LFS performance decrease after several hours of
> > intensive load. Unfortunately, that's how SSD drives work:
> > http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-arc
> > hitecture-of-an-ssd-and-benchmarking/
> > Consider buying fast production-level SSD drives.
> > 2) Consider using separate drives for LFS files and WAL. Ignite actively
> > performs writes to both LFS and WAL under intensive load, and having two
> > devices will double your throughput limit.
> > 3) Over-provision your SSD. Performance of random writes on 50% filled
> > disk is much better than on 90% filled. SSD Over-Provisioning And Its
> > Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisio
> > ning-benefits-master-ti/
> > 4) Leave free space in RAM to let OS use page cache and optimize writes.
> > Total size of all memory policies shouldn't exceed 70% of your RAM.
> > 5) Make sure that OS doesn't utilize swap. If you use Unix, best option
> is
> > set vm.swappiness to 0.
> > 6) Try to find out page size of your SSD. Ideally, page size of Ignite
> > shouldn't be less than SSD page size. Possible approaches:
> > Find it in device specification (some manufacturers don't reveal it)
> > Try running SSD benchmarks
> > If you are not sure, just set page size to 4K. As various benchmarks use
> > 4K pages, manufacturers have to adapt drives for 4K random write
> workload.
> > Whitepaper from Intel showing that 4K pages are enough:
> > https://www.intel.com/content/dam/www/public/us/en/documents
> > /white-papers/ssd-server-storage-applications-paper.pdf
> > Check your OS page cache size. Page size of Ignite shouldn't be less than
> > OS page size. How to check OS cache page size in Unix:
> > https://unix.stackexchange.com/questions/128213/how-is-page-
> > size-determined-in-virtual-address-space
> >
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 01.09.2017 21:08, Denis Magda wrote:
> >
> >> Igniters,
> >>
> >> I see a lot of complains regarding the performance of the subj on the
> >> user list. At the same time, I do believe that in most scenarios it’s a
> >> lack of knowledge that we keep in secret.
> >>
> >> It's time to document Durable Memory and its Native Persistence tuning
> >> parameters. Let's start doing this for Linux based deployments first.
> Here
> >> is what we have for now (which is almost nothing):
> >> https://apacheignite.readme.io/docs/durable-memory-tuning <
> >> https://apacheignite.readme.io/docs/durable-memory-tuning>
> >>
> >> Ideally, at some point we have to come up with doc like this:
> >> https://access.redhat.com/sites/default/files/attachments/
> >> deploying-oracle-12c-on-rhel6_1.2_1.pdf
> >>
> >> Please share your expertise in a form of settings that have to be put on
> >> the paper. We put them in JIRA and document afterwords:
> >> https://issues.apache.org/jira/browse/IGNITE-6246 <
> >> https://issues.apache.org/jira/browse/IGNITE-6246>
> >>
> >> —
> >> Denis
> >>
> >
> >
>

[dmagda]

That’s exactly the discussion around that documentation. Feel free to add these useful points there or wait while I’ll do this later.

—
Denis

> On Sep 13, 2017, at 12:27 PM, Dmitriy Setrakyan <[hidden email]> wrote:
>
> Completely agree with Nikita. Why not add this information here?
>
> https://apacheignite.readme.io/docs/durable-memory-tuning
>
> D.
>
> On Wed, Sep 13, 2017 at 2:54 AM, Nikita Ivanov <[hidden email]> wrote:
>
>> Excellent work on this... This should be expanded and be prominently placed
>> in our docs/tutorials/javadocs/etc.
>>
>> --
>> Nikita Ivanov
>>
>>
>> On Wed, Sep 13, 2017 at 12:29 PM, Ivan Rakov <[hidden email]>
>> wrote:
>>
>>> Folks,
>>>
>>> We had some experience of benchmarking Ignite with persistent store on
>>> SSD. I think we can share some helpful advice. None of them require
>>> changing configuration of Ignite or persistent store.
>>>
>>> *Tuning advice for users*
>>>
>>> 1) Be prepared for LFS performance decrease after several hours of
>>> intensive load. Unfortunately, that's how SSD drives work:
>>> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-arc
>>> hitecture-of-an-ssd-and-benchmarking/
>>> Consider buying fast production-level SSD drives.
>>> 2) Consider using separate drives for LFS files and WAL. Ignite actively
>>> performs writes to both LFS and WAL under intensive load, and having two
>>> devices will double your throughput limit.
>>> 3) Over-provision your SSD. Performance of random writes on 50% filled
>>> disk is much better than on 90% filled. SSD Over-Provisioning And Its
>>> Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisio
>>> ning-benefits-master-ti/
>>> 4) Leave free space in RAM to let OS use page cache and optimize writes.
>>> Total size of all memory policies shouldn't exceed 70% of your RAM.
>>> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option
>> is
>>> set vm.swappiness to 0.
>>> 6) Try to find out page size of your SSD. Ideally, page size of Ignite
>>> shouldn't be less than SSD page size. Possible approaches:
>>> Find it in device specification (some manufacturers don't reveal it)
>>> Try running SSD benchmarks
>>> If you are not sure, just set page size to 4K. As various benchmarks use
>>> 4K pages, manufacturers have to adapt drives for 4K random write
>> workload.
>>> Whitepaper from Intel showing that 4K pages are enough:
>>> https://www.intel.com/content/dam/www/public/us/en/documents
>>> /white-papers/ssd-server-storage-applications-paper.pdf
>>> Check your OS page cache size. Page size of Ignite shouldn't be less than
>>> OS page size. How to check OS cache page size in Unix:
>>> https://unix.stackexchange.com/questions/128213/how-is-page-
>>> size-determined-in-virtual-address-space
>>>
>>>
>>> Best Regards,
>>> Ivan Rakov
>>>
>>> On 01.09.2017 21:08, Denis Magda wrote:
>>>
>>>> Igniters,
>>>>
>>>> I see a lot of complains regarding the performance of the subj on the
>>>> user list. At the same time, I do believe that in most scenarios it’s a
>>>> lack of knowledge that we keep in secret.
>>>>
>>>> It's time to document Durable Memory and its Native Persistence tuning
>>>> parameters. Let's start doing this for Linux based deployments first.
>> Here
>>>> is what we have for now (which is almost nothing):
>>>> https://apacheignite.readme.io/docs/durable-memory-tuning <
>>>> https://apacheignite.readme.io/docs/durable-memory-tuning>
>>>>
>>>> Ideally, at some point we have to come up with doc like this:
>>>> https://access.redhat.com/sites/default/files/attachments/
>>>> deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>>>
>>>> Please share your expertise in a form of settings that have to be put on
>>>> the paper. We put them in JIRA and document afterwords:
>>>> https://issues.apache.org/jira/browse/IGNITE-6246 <
>>>> https://issues.apache.org/jira/browse/IGNITE-6246>
>>>>
>>>> —
>>>> Denis
>>>>
>>>
>>>
>>

[dmagda]

Ivan,

Documented:
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning>

However, I’m a bit confused with you 6. point regarding the page size calculation. Should Ignite page size be always less than the SSD’s page size and OS page cache size? Or it can be equal? For instance if the OS page cache size is 4 KB what should be Ignite’s page size? That’s the section about this: https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size>

—
Denis

> On Sep 13, 2017, at 2:29 AM, Ivan Rakov <[hidden email]> wrote:
>
> Folks,
>
> We had some experience of benchmarking Ignite with persistent store on SSD. I think we can share some helpful advice. None of them require changing configuration of Ignite or persistent store.
>
> *Tuning advice for users*
>
> 1) Be prepared for LFS performance decrease after several hours of intensive load. Unfortunately, that's how SSD drives work: http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
> Consider buying fast production-level SSD drives.
> 2) Consider using separate drives for LFS files and WAL. Ignite actively performs writes to both LFS and WAL under intensive load, and having two devices will double your throughput limit.
> 3) Over-provision your SSD. Performance of random writes on 50% filled disk is much better than on 90% filled. SSD Over-Provisioning And Its Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
> 4) Leave free space in RAM to let OS use page cache and optimize writes. Total size of all memory policies shouldn't exceed 70% of your RAM.
> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is set vm.swappiness to 0.
> 6) Try to find out page size of your SSD. Ideally, page size of Ignite shouldn't be less than SSD page size. Possible approaches:
> Find it in device specification (some manufacturers don't reveal it)
> Try running SSD benchmarks
> If you are not sure, just set page size to 4K. As various benchmarks use 4K pages, manufacturers have to adapt drives for 4K random write workload. Whitepaper from Intel showing that 4K pages are enough: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
> Check your OS page cache size. Page size of Ignite shouldn't be less than OS page size. How to check OS cache page size in Unix: https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
>
>
> Best Regards,
> Ivan Rakov
>
> On 01.09.2017 21:08, Denis Magda wrote:
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the user list. At the same time, I do believe that in most scenarios it’s a lack of knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence tuning parameters. Let's start doing this for Linux based deployments first. Here is what we have for now (which is almost nothing): https://apacheignite.readme.io/docs/durable-memory-tuning <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>
>> Ideally, at some point we have to come up with doc like this: https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put on the paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246 <https://issues.apache.org/jira/browse/IGNITE-6246>
>>
>> —
>> Denis
>

[dmagda]

Igniters,

The first version of the guide is ready. Please check it up before it’s shared with the users:
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning

*Ivan*, please clarify this:

- Should Ignite page size be always less than the SSD’s page size and OS page cache size? Or it can be *equal* as well? For instance if the OS page cache size is 4 KB what should be Ignite’s page size? I would set 4 which is not less! Check this paragraph here [1]

- It’s said that Intel confirm 4 KB should be enough if you can’t sort out SSD page size value and you give this link [2]. Please point out to the page where this is stated.

[1] https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size
[2] https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf

—
Denis

> On Sep 13, 2017, at 5:46 PM, Denis Magda <[hidden email]> wrote:
>
> Ivan,
>
> Documented:
> https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning>
>
> However, I’m a bit confused with you 6. point regarding the page size calculation. Should Ignite page size be always less than the SSD’s page size and OS page cache size? Or it can be equal? For instance if the OS page cache size is 4 KB what should be Ignite’s page size? That’s the section about this: https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size>
>
> —
> Denis
>
>> On Sep 13, 2017, at 2:29 AM, Ivan Rakov <[hidden email]> wrote:
>>
>> Folks,
>>
>> We had some experience of benchmarking Ignite with persistent store on SSD. I think we can share some helpful advice. None of them require changing configuration of Ignite or persistent store.
>>
>> *Tuning advice for users*
>>
>> 1) Be prepared for LFS performance decrease after several hours of intensive load. Unfortunately, that's how SSD drives work: http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
>> Consider buying fast production-level SSD drives.
>> 2) Consider using separate drives for LFS files and WAL. Ignite actively performs writes to both LFS and WAL under intensive load, and having two devices will double your throughput limit.
>> 3) Over-provision your SSD. Performance of random writes on 50% filled disk is much better than on 90% filled. SSD Over-Provisioning And Its Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
>> 4) Leave free space in RAM to let OS use page cache and optimize writes. Total size of all memory policies shouldn't exceed 70% of your RAM.
>> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is set vm.swappiness to 0.
>> 6) Try to find out page size of your SSD. Ideally, page size of Ignite shouldn't be less than SSD page size. Possible approaches:
>> Find it in device specification (some manufacturers don't reveal it)
>> Try running SSD benchmarks
>> If you are not sure, just set page size to 4K. As various benchmarks use 4K pages, manufacturers have to adapt drives for 4K random write workload. Whitepaper from Intel showing that 4K pages are enough: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
>> Check your OS page cache size. Page size of Ignite shouldn't be less than OS page size. How to check OS cache page size in Unix: https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
>>
>>
>> Best Regards,
>> Ivan Rakov
>>
>> On 01.09.2017 21:08, Denis Magda wrote:
>>> Igniters,
>>>
>>> I see a lot of complains regarding the performance of the subj on the user list. At the same time, I do believe that in most scenarios it’s a lack of knowledge that we keep in secret.
>>>
>>> It's time to document Durable Memory and its Native Persistence tuning parameters. Let's start doing this for Linux based deployments first. Here is what we have for now (which is almost nothing): https://apacheignite.readme.io/docs/durable-memory-tuning <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>>
>>> Ideally, at some point we have to come up with doc like this: https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>>
>>> Please share your expertise in a form of settings that have to be put on the paper. We put them in JIRA and document afterwords:
>>> https://issues.apache.org/jira/browse/IGNITE-6246 <https://issues.apache.org/jira/browse/IGNITE-6246>
>>>
>>> —
>>> Denis
>>
>

[Ivan Rakov]

Denis,

Yes, Ignite page size should be *greater or equal* than OS page cache
size and SSD page size. I mentioned it in advice list:

> page size of Ignite shouldn't be less than SSD page size
> Page size of Ignite shouldn't be less than OS page size

Sorry for vague wording.
We should fix this in documentation.


Best Regards,
Ivan Rakov

On 14.09.2017 3:46, Denis Magda wrote:

> Ivan,
>
> Documented:
> https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning>
>
> However, I’m a bit confused with you 6. point regarding the page size calculation. Should Ignite page size be always less than the SSD’s page size and OS page cache size? Or it can be equal? For instance if the OS page cache size is 4 KB what should be Ignite’s page size? That’s the section about this: https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size>
>
> —
> Denis
>
>> On Sep 13, 2017, at 2:29 AM, Ivan Rakov <[hidden email]> wrote:
>>
>> Folks,
>>
>> We had some experience of benchmarking Ignite with persistent store on SSD. I think we can share some helpful advice. None of them require changing configuration of Ignite or persistent store.
>>
>> *Tuning advice for users*
>>
>> 1) Be prepared for LFS performance decrease after several hours of intensive load. Unfortunately, that's how SSD drives work: http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
>> Consider buying fast production-level SSD drives.
>> 2) Consider using separate drives for LFS files and WAL. Ignite actively performs writes to both LFS and WAL under intensive load, and having two devices will double your throughput limit.
>> 3) Over-provision your SSD. Performance of random writes on 50% filled disk is much better than on 90% filled. SSD Over-Provisioning And Its Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
>> 4) Leave free space in RAM to let OS use page cache and optimize writes. Total size of all memory policies shouldn't exceed 70% of your RAM.
>> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is set vm.swappiness to 0.
>> 6) Try to find out page size of your SSD. Ideally, page size of Ignite shouldn't be less than SSD page size. Possible approaches:
>> Find it in device specification (some manufacturers don't reveal it)
>> Try running SSD benchmarks
>> If you are not sure, just set page size to 4K. As various benchmarks use 4K pages, manufacturers have to adapt drives for 4K random write workload. Whitepaper from Intel showing that 4K pages are enough: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
>> Check your OS page cache size. Page size of Ignite shouldn't be less than OS page size. How to check OS cache page size in Unix: https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
>>
>>
>> Best Regards,
>> Ivan Rakov
>>
>> On 01.09.2017 21:08, Denis Magda wrote:
>>> Igniters,
>>>
>>> I see a lot of complains regarding the performance of the subj on the user list. At the same time, I do believe that in most scenarios it’s a lack of knowledge that we keep in secret.
>>>
>>> It's time to document Durable Memory and its Native Persistence tuning parameters. Let's start doing this for Linux based deployments first. Here is what we have for now (which is almost nothing): https://apacheignite.readme.io/docs/durable-memory-tuning <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>>
>>> Ideally, at some point we have to come up with doc like this: https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>>
>>> Please share your expertise in a form of settings that have to be put on the paper. We put them in JIRA and document afterwords:
>>> https://issues.apache.org/jira/browse/IGNITE-6246 <https://issues.apache.org/jira/browse/IGNITE-6246>
>>>
>>> —
>>> Denis
>

[dmagda]

Thanks Ivan, *no less* sounds best for me.

Prachi, please do final editing of the doc:
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning

—
Denis

> On Sep 15, 2017, at 12:24 AM, Ivan Rakov <[hidden email]> wrote:
>
> Denis,
>
> Yes, Ignite page size should be *greater or equal* than OS page cache size and SSD page size. I mentioned it in advice list:
>
>> page size of Ignite shouldn't be less than SSD page size
>> Page size of Ignite shouldn't be less than OS page size
>
> Sorry for vague wording.
> We should fix this in documentation.
>
>
> Best Regards,
> Ivan Rakov
>
> On 14.09.2017 3:46, Denis Magda wrote:
>> Ivan,
>>
>> Documented:
>> https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning>
>>
>> However, I’m a bit confused with you 6. point regarding the page size calculation. Should Ignite page size be always less than the SSD’s page size and OS page cache size? Or it can be equal? For instance if the OS page cache size is 4 KB what should be Ignite’s page size? That’s the section about this: https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size <https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size>
>>
>> —
>> Denis
>>
>>> On Sep 13, 2017, at 2:29 AM, Ivan Rakov <[hidden email]> wrote:
>>>
>>> Folks,
>>>
>>> We had some experience of benchmarking Ignite with persistent store on SSD. I think we can share some helpful advice. None of them require changing configuration of Ignite or persistent store.
>>>
>>> *Tuning advice for users*
>>>
>>> 1) Be prepared for LFS performance decrease after several hours of intensive load. Unfortunately, that's how SSD drives work: http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
>>> Consider buying fast production-level SSD drives.
>>> 2) Consider using separate drives for LFS files and WAL. Ignite actively performs writes to both LFS and WAL under intensive load, and having two devices will double your throughput limit.
>>> 3) Over-provision your SSD. Performance of random writes on 50% filled disk is much better than on 90% filled. SSD Over-Provisioning And Its Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
>>> 4) Leave free space in RAM to let OS use page cache and optimize writes. Total size of all memory policies shouldn't exceed 70% of your RAM.
>>> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is set vm.swappiness to 0.
>>> 6) Try to find out page size of your SSD. Ideally, page size of Ignite shouldn't be less than SSD page size. Possible approaches:
>>> Find it in device specification (some manufacturers don't reveal it)
>>> Try running SSD benchmarks
>>> If you are not sure, just set page size to 4K. As various benchmarks use 4K pages, manufacturers have to adapt drives for 4K random write workload. Whitepaper from Intel showing that 4K pages are enough: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
>>> Check your OS page cache size. Page size of Ignite shouldn't be less than OS page size. How to check OS cache page size in Unix: https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
>>>
>>>
>>> Best Regards,
>>> Ivan Rakov
>>>
>>> On 01.09.2017 21:08, Denis Magda wrote:
>>>> Igniters,
>>>>
>>>> I see a lot of complains regarding the performance of the subj on the user list. At the same time, I do believe that in most scenarios it’s a lack of knowledge that we keep in secret.
>>>>
>>>> It's time to document Durable Memory and its Native Persistence tuning parameters. Let's start doing this for Linux based deployments first. Here is what we have for now (which is almost nothing): https://apacheignite.readme.io/docs/durable-memory-tuning <https://apacheignite.readme.io/docs/durable-memory-tuning>
>>>>
>>>> Ideally, at some point we have to come up with doc like this: https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>>>
>>>> Please share your expertise in a form of settings that have to be put on the paper. We put them in JIRA and document afterwords:
>>>> https://issues.apache.org/jira/browse/IGNITE-6246 <https://issues.apache.org/jira/browse/IGNITE-6246>
>>>>
>>>> —
>>>> Denis
>>
>