Default page size must be changed to 4k. Should it be backwards compatible?

[Ivan Rakov]

Guys,

We have benchmarked how checkpoint write speed on SSD disk depends on
various parameters. It became absolutely obvious that using 4K pages in
durable memory instead of 2K brings considerable, significant speed-up.
I think, we must set 4K as default page size.
Ticket with detailed explanation:
https://issues.apache.org/jira/browse/IGNITE-5884
Spoiler: it depends on write order and alignment, but writing 4K is at
least *3x faster* than writing 2K when other parameters are the same.

The question is backwards compatibility. If pageSize is not explicitly
set in user configuration, attempt to start "4k default" Ignite node
from "2k default" LFS files will fail with exception:

> class org.apache.ignite.IgniteCheckedException: Failed to verify store
> file (invalid page size) [expectedPageSize=4096, filePageSize=2048]
>     at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.checkFile(FilePageStore.java:206)
>     at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:416)
>     at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:315)
>     at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:287)
>     at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:272)
>     at
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569)
>     at
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487)
>     at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.getOrAllocateCacheMetas(GridCacheOffheapManager.java:515)
>     at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.initDataStructures(GridCacheOffheapManager.java:86)
>     at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.start(IgniteCacheOffheapManagerImpl.java:139)
>     at
> org.apache.ignite.internal.processors.cache.CacheGroupContext.start(CacheGroupContext.java:868)

I think, we have two options here:

1) Obvious and safe - provide silent backwards compatibility. We can
implement a task which will find any LFS file, check its pageSize and
use it as default.
2) Less user-friendly, but in my opinion still better option - crash
node, but make error message more informative. We'll let user know that
default pageSize was changed to 4k due to discovered performance boost
on most UNIX-based enviroments with SSD (which is for sure most popular
enviroment among users), and recommend user to migrate to 4K-page LFS.
If user still wants to work with 2k pages, he can always set it
explicitly in MemoryConfiguration and start node.

Thoughts?

--
Best Regards,
Ivan Rakov

[dsetrakyan]

I like #1 if possible. Of course, if the LFS is empty, then the new default
should be 4k.

On Tue, Aug 15, 2017 at 12:01 PM, Ivan Rakov <[hidden email]> wrote:

> Guys,
>
> We have benchmarked how checkpoint write speed on SSD disk depends on
> various parameters. It became absolutely obvious that using 4K pages in
> durable memory instead of 2K brings considerable, significant speed-up. I
> think, we must set 4K as default page size.
> Ticket with detailed explanation: https://issues.apache.org/jira
> /browse/IGNITE-5884
> Spoiler: it depends on write order and alignment, but writing 4K is at
> least *3x faster* than writing 2K when other parameters are the same.
>
> The question is backwards compatibility. If pageSize is not explicitly set
> in user configuration, attempt to start "4k default" Ignite node from "2k
> default" LFS files will fail with exception:
>
> class org.apache.ignite.IgniteCheckedException: Failed to verify store
>> file (invalid page size) [expectedPageSize=4096, filePageSize=2048]
>>     at org.apache.ignite.internal.processors.cache.persistence.file
>> .FilePageStore.checkFile(FilePageStore.java:206)
>>     at org.apache.ignite.internal.processors.cache.persistence.file
>> .FilePageStore.init(FilePageStore.java:416)
>>     at org.apache.ignite.internal.processors.cache.persistence.file
>> .FilePageStore.read(FilePageStore.java:315)
>>     at org.apache.ignite.internal.processors.cache.persistence.file
>> .FilePageStoreManager.read(FilePageStoreManager.java:287)
>>     at org.apache.ignite.internal.processors.cache.persistence.file
>> .FilePageStoreManager.read(FilePageStoreManager.java:272)
>>     at org.apache.ignite.internal.processors.cache.persistence.page
>> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569)
>>     at org.apache.ignite.internal.processors.cache.persistence.page
>> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487)
>>     at org.apache.ignite.internal.processors.cache.persistence.Grid
>> CacheOffheapManager.getOrAllocateCacheMetas(GridCacheOffheap
>> Manager.java:515)
>>     at org.apache.ignite.internal.processors.cache.persistence.Grid
>> CacheOffheapManager.initDataStructures(GridCacheOffheapManager.java:86)
>>     at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.start(IgniteCacheOffheapManagerImpl.java:139)
>>     at org.apache.ignite.internal.processors.cache.CacheGroupContex
>> t.start(CacheGroupContext.java:868)
>>
>
> I think, we have two options here:
>
> 1) Obvious and safe - provide silent backwards compatibility. We can
> implement a task which will find any LFS file, check its pageSize and use
> it as default.
> 2) Less user-friendly, but in my opinion still better option - crash node,
> but make error message more informative. We'll let user know that default
> pageSize was changed to 4k due to discovered performance boost on most
> UNIX-based enviroments with SSD (which is for sure most popular enviroment
> among users), and recommend user to migrate to 4K-page LFS. If user still
> wants to work with 2k pages, he can always set it explicitly in
> MemoryConfiguration and start node.
>
> Thoughts?
>
> --
> Best Regards,
> Ivan Rakov
>
>

[Serge Puchnin]

CONTENTS DELETED

The author has deleted this message.

[dsetrakyan]

On Tue, Aug 15, 2017 at 10:09 PM, Serge Puchnin <[hidden email]>
wrote:

> #1 option looks more predictable. But it's possible to add a message "for
> better IO performance please migrate to 4K-pages".
>

Agree. Can we update the ticket to make sure that this suggestion is
printed out?

> BR,
> Serge
>
>
> On Wed, 16 Aug 2017 at 03:46, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > I like #1 if possible. Of course, if the LFS is empty, then the new
> default
> > should be 4k.
> >
> > On Tue, Aug 15, 2017 at 12:01 PM, Ivan Rakov <[hidden email]>
> > wrote:
> >
> > > Guys,
> > >
> > > We have benchmarked how checkpoint write speed on SSD disk depends on
> > > various parameters. It became absolutely obvious that using 4K pages in
> > > durable memory instead of 2K brings considerable, significant
> speed-up. I
> > > think, we must set 4K as default page size.
> > > Ticket with detailed explanation: https://issues.apache.org/jira
> > > /browse/IGNITE-5884
> > > Spoiler: it depends on write order and alignment, but writing 4K is at
> > > least *3x faster* than writing 2K when other parameters are the same.
> > >
> > > The question is backwards compatibility. If pageSize is not explicitly
> > set
> > > in user configuration, attempt to start "4k default" Ignite node from
> "2k
> > > default" LFS files will fail with exception:
> > >
> > > class org.apache.ignite.IgniteCheckedException: Failed to verify store
> > >> file (invalid page size) [expectedPageSize=4096, filePageSize=2048]
> > >>     at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStore.checkFile(FilePageStore.java:206)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStore.init(FilePageStore.java:416)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStore.read(FilePageStore.java:315)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStoreManager.read(FilePageStoreManager.java:287)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStoreManager.read(FilePageStoreManager.java:272)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.page
> > >> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.page
> > >> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.Grid
> > >> CacheOffheapManager.getOrAllocateCacheMetas(GridCacheOffheap
> > >> Manager.java:515)
> > >>     at org.apache.ignite.internal.processors.cache.persistence.Grid
> > >> CacheOffheapManager.initDataStructures(GridCacheOffheapManager.java:
> 86)
> > >>     at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> > >> apManagerImpl.start(IgniteCacheOffheapManagerImpl.java:139)
> > >>     at org.apache.ignite.internal.processors.cache.CacheGroupContex
> > >> t.start(CacheGroupContext.java:868)
> > >>
> > >
> > > I think, we have two options here:
> > >
> > > 1) Obvious and safe - provide silent backwards compatibility. We can
> > > implement a task which will find any LFS file, check its pageSize and
> use
> > > it as default.
> > > 2) Less user-friendly, but in my opinion still better option - crash
> > node,
> > > but make error message more informative. We'll let user know that
> default
> > > pageSize was changed to 4k due to discovered performance boost on most
> > > UNIX-based enviroments with SSD (which is for sure most popular
> > enviroment
> > > among users), and recommend user to migrate to 4K-page LFS. If user
> still
> > > wants to work with 2k pages, he can always set it explicitly in
> > > MemoryConfiguration and start node.
> > >
> > > Thoughts?
> > >
> > > --
> > > Best Regards,
> > > Ivan Rakov
> > >
> > >
> >
>

[dmagda]

Added Ignite Persistence related section to the “Performance Tips” docs and mentioned this suggestion there:
https://apacheignite.readme.io/docs/performance-tips#tune-ignite-native-persistence

—
Denis

> On Aug 17, 2017, at 6:18 PM, Dmitriy Setrakyan <[hidden email]> wrote:
>
> On Tue, Aug 15, 2017 at 10:09 PM, Serge Puchnin <[hidden email]>
> wrote:
>
>> #1 option looks more predictable. But it's possible to add a message "for
>> better IO performance please migrate to 4K-pages".
>>
>
> Agree. Can we update the ticket to make sure that this suggestion is
> printed out?
>
>
>> BR,
>> Serge
>>
>>
>> On Wed, 16 Aug 2017 at 03:46, Dmitriy Setrakyan <[hidden email]>
>> wrote:
>>
>>> I like #1 if possible. Of course, if the LFS is empty, then the new
>> default
>>> should be 4k.
>>>
>>> On Tue, Aug 15, 2017 at 12:01 PM, Ivan Rakov <[hidden email]>
>>> wrote:
>>>
>>>> Guys,
>>>>
>>>> We have benchmarked how checkpoint write speed on SSD disk depends on
>>>> various parameters. It became absolutely obvious that using 4K pages in
>>>> durable memory instead of 2K brings considerable, significant
>> speed-up. I
>>>> think, we must set 4K as default page size.
>>>> Ticket with detailed explanation: https://issues.apache.org/jira
>>>> /browse/IGNITE-5884
>>>> Spoiler: it depends on write order and alignment, but writing 4K is at
>>>> least *3x faster* than writing 2K when other parameters are the same.
>>>>
>>>> The question is backwards compatibility. If pageSize is not explicitly
>>> set
>>>> in user configuration, attempt to start "4k default" Ignite node from
>> "2k
>>>> default" LFS files will fail with exception:
>>>>
>>>> class org.apache.ignite.IgniteCheckedException: Failed to verify store
>>>>> file (invalid page size) [expectedPageSize=4096, filePageSize=2048]
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.file
>>>>> .FilePageStore.checkFile(FilePageStore.java:206)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.file
>>>>> .FilePageStore.init(FilePageStore.java:416)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.file
>>>>> .FilePageStore.read(FilePageStore.java:315)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.file
>>>>> .FilePageStoreManager.read(FilePageStoreManager.java:287)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.file
>>>>> .FilePageStoreManager.read(FilePageStoreManager.java:272)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.page
>>>>> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.page
>>>>> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.Grid
>>>>> CacheOffheapManager.getOrAllocateCacheMetas(GridCacheOffheap
>>>>> Manager.java:515)
>>>>>    at org.apache.ignite.internal.processors.cache.persistence.Grid
>>>>> CacheOffheapManager.initDataStructures(GridCacheOffheapManager.java:
>> 86)
>>>>>    at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>>>>> apManagerImpl.start(IgniteCacheOffheapManagerImpl.java:139)
>>>>>    at org.apache.ignite.internal.processors.cache.CacheGroupContex
>>>>> t.start(CacheGroupContext.java:868)
>>>>>
>>>>
>>>> I think, we have two options here:
>>>>
>>>> 1) Obvious and safe - provide silent backwards compatibility. We can
>>>> implement a task which will find any LFS file, check its pageSize and
>> use
>>>> it as default.
>>>> 2) Less user-friendly, but in my opinion still better option - crash
>>> node,
>>>> but make error message more informative. We'll let user know that
>> default
>>>> pageSize was changed to 4k due to discovered performance boost on most
>>>> UNIX-based enviroments with SSD (which is for sure most popular
>>> enviroment
>>>> among users), and recommend user to migrate to 4K-page LFS. If user
>> still
>>>> wants to work with 2k pages, he can always set it explicitly in
>>>> MemoryConfiguration and start node.
>>>>
>>>> Thoughts?
>>>>
>>>> --
>>>> Best Regards,
>>>> Ivan Rakov
>>>>
>>>>
>>>
>>