Apache Ignite Developers - Legacy Mail Archive

Ignite Durable Memory & Persistent Store introduction for developers

Classic

List

Threaded

15 messages Options

Dmitriy Pavlov

Ignite Durable Memory & Persistent Store introduction for developers

Hi Igniters,

I am glad to introduce two new wiki articles. These articles contain
high-level description of

- Ignite Durable Memory (
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
)

- and Ignite Persistent Store (
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
)

These articles cover basic principles and internal design. Pages may be
useful for new contributors during start of working with persistent store
code base.

I would like to thank all contributors who helped me with my questions.
Their answers and support made it possible to start this description.

I will appreciate any feedback and questions. Suggestions and edits are
welcome, of course.

Sincerely,

Dmitriy Pavlov

Anton Vinogradov

Re: Ignite Durable Memory & Persistent Store introduction for developers

Awesome!
Thanks!

On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]>
wrote:

> Hi Igniters,
>
>
> I am glad to introduce two new wiki articles. These articles contain
> high-level description of
>
> - Ignite Durable Memory (
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Durable+Memory+-+under+the+hood
> )
>
> - and Ignite Persistent Store (
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Persistent+Store+-+under+the+hood
> )
>
> These articles cover basic principles and internal design. Pages may be
> useful for new contributors during start of working with persistent store
> code base.
>
>
> I would like to thank all contributors who helped me with my questions.
> Their answers and support made it possible to start this description.
>
>
> I will appreciate any feedback and questions. Suggestions and edits are
> welcome, of course.
>
>
> Sincerely,
>
> Dmitriy Pavlov
>

dsetrakyan

Re: Ignite Durable Memory & Persistent Store introduction for developers

In reply to this post by Dmitriy Pavlov

Dmitriy, this is very useful! Any chance you can also blog about it?

On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]>
wrote:

Dmitriy Pavlov

Re: Ignite Durable Memory & Persistent Store introduction for developers

Sure, I can blog about it, but timelines depend on free time. Which
language is preferred? Russian or English? Also I plan to extend existing
wiki pages with more details, and add new pages explaining Ignite to
newbies. This activity depends on interest of community and new questions
related to pages.

чт, 31 авг. 2017 г. в 3:53, Dmitriy Setrakyan <[hidden email]>:

> Dmitriy, this is very useful! Any chance you can also blog about it?
>
> On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Hi Igniters,
> >
> >
> > I am glad to introduce two new wiki articles. These articles contain
> > high-level description of
> >
> > - Ignite Durable Memory (
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Durable+Memory+-+under+the+hood
> > )
> >
> > - and Ignite Persistent Store (
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Persistent+Store+-+under+the+hood
> > )
> >
> > These articles cover basic principles and internal design. Pages may be
> > useful for new contributors during start of working with persistent store
> > code base.
> >
> >
> > I would like to thank all contributors who helped me with my questions.
> > Their answers and support made it possible to start this description.
> >
> >
> > I will appreciate any feedback and questions. Suggestions and edits are
> > welcome, of course.
> >
> >
> > Sincerely,
> >
> > Dmitriy Pavlov
> >
>

Alexey Dmitriev

Re: Ignite Durable Memory & Persistent Store introduction for developers

Dmitry,
Both languages are preferable. :)
English has priority.

31 авг. 2017 г. 1:06 ПП пользователь "Dmitry Pavlov" <[hidden email]>
написал:

> Sure, I can blog about it, but timelines depend on free time. Which
> language is preferred? Russian or English? Also I plan to extend existing
> wiki pages with more details, and add new pages explaining Ignite to
> newbies. This activity depends on interest of community and new questions
> related to pages.
>
> чт, 31 авг. 2017 г. в 3:53, Dmitriy Setrakyan <[hidden email]>:
>
> > Dmitriy, this is very useful! Any chance you can also blog about it?
> >
> > On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Hi Igniters,
> > >
> > >
> > > I am glad to introduce two new wiki articles. These articles contain
> > > high-level description of
> > >
> > > - Ignite Durable Memory (
> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > Ignite+Durable+Memory+-+under+the+hood
> > > )
> > >
> > > - and Ignite Persistent Store (
> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > Ignite+Persistent+Store+-+under+the+hood
> > > )
> > >
> > > These articles cover basic principles and internal design. Pages may be
> > > useful for new contributors during start of working with persistent
> store
> > > code base.
> > >
> > >
> > > I would like to thank all contributors who helped me with my questions.
> > > Their answers and support made it possible to start this description.
> > >
> > >
> > > I will appreciate any feedback and questions. Suggestions and edits are
> > > welcome, of course.
> > >
> > >
> > > Sincerely,
> > >
> > > Dmitriy Pavlov
> > >
> >
>

李玉珏@163

Re: Ignite Durable Memory & Persistent Store introduction for developers

I will help to translate into chinese.

在 2017/8/31 下午6:39, Alexey Dmitriev 写道:

> Dmitry,
> Both languages are preferable. :)
> English has priority.
>
> 31 авг. 2017 г. 1:06 ПП пользователь "Dmitry Pavlov" <[hidden email]>
> написал:
>
>> Sure, I can blog about it, but timelines depend on free time. Which
>> language is preferred? Russian or English? Also I plan to extend existing
>> wiki pages with more details, and add new pages explaining Ignite to
>> newbies. This activity depends on interest of community and new questions
>> related to pages.
>>
>> чт, 31 авг. 2017 г. в 3:53, Dmitriy Setrakyan <[hidden email]>:
>>
>>> Dmitriy, this is very useful! Any chance you can also blog about it?
>>>
>>> On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]>
>>> wrote:
>>>
>>>> Hi Igniters,
>>>>
>>>>
>>>> I am glad to introduce two new wiki articles. These articles contain
>>>> high-level description of
>>>>
>>>> - Ignite Durable Memory (
>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>> Ignite+Durable+Memory+-+under+the+hood
>>>> )
>>>>
>>>> - and Ignite Persistent Store (
>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>> Ignite+Persistent+Store+-+under+the+hood
>>>> )
>>>>
>>>> These articles cover basic principles and internal design. Pages may be
>>>> useful for new contributors during start of working with persistent
>> store
>>>> code base.
>>>>
>>>>
>>>> I would like to thank all contributors who helped me with my questions.
>>>> Their answers and support made it possible to start this description.
>>>>
>>>>
>>>> I will appreciate any feedback and questions. Suggestions and edits are
>>>> welcome, of course.
>>>>
>>>>
>>>> Sincerely,
>>>>
>>>> Dmitriy Pavlov
>>>>

dmagda

Re: Ignite Durable Memory & Persistent Store introduction for developers

In reply to this post by Dmitriy Pavlov

Dmitriy,

First of all, thanks for putting this low level details on paper! Current and future community members will be grateful to you.

Second, let me be a reviewer of the page and suggest some improvements. Until the review is over I would encourage us to hold on with any public announcements such as blog post.

I’ve reviewed and edited Ignite Durable Memory section (https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory <https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>). Please address the questions below and I’ll move on with the new sections afterwards.

1) "Different caches and its partitions bring more complex addressing scheme but it is still possible to map from page ID to position in particular file: pageId => file + offset in this file”.

Probably this appears too early in the doc because above we just introduced page ID notion with durable memory and now we’re talking about some files? Are these files of ignite persistence? What if the persistence is disabled? Honestly, the sentence gets out of the story context.

2) "Also, the tree pages can be merged into one if < 50% of space is used.”

50% of the whole memory available? Please clarify.

3) "There is one page ID table per memory segment.”

It’s the same segment as it’s described in the main doc, right? https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment <https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment>

4) "within unsafe segment”

If we want to mention “usafe” here we need to explain what that is.

5) "If memory amount less than whole index size, Ignite still can operate.”

How do we define “index size”? Is this the page size?

—
Denis

> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> wrote:
>
> Hi Igniters,
>
>
> I am glad to introduce two new wiki articles. These articles contain
> high-level description of
>
> - Ignite Durable Memory (
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> )
>
> - and Ignite Persistent Store (
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> )
>
> These articles cover basic principles and internal design. Pages may be
> useful for new contributors during start of working with persistent store
> code base.
>
>
> I would like to thank all contributors who helped me with my questions.
> Their answers and support made it possible to start this description.
>
>
> I will appreciate any feedback and questions. Suggestions and edits are
> welcome, of course.
>
>
> Sincerely,
>
> Dmitriy Pavlov

Dmitriy Pavlov

Re: Ignite Durable Memory & Persistent Store introduction for developers

Hi Denis,

Sure, I would be glad if you have some time to review the page. Thank you
for edits and careful review!

According to recent questions and feedback from Igniters, I would also like
to extend section about memory organization.

Please give me a couple of days to address your questions and extend wiki.

Sincerely,
Dmitriy Pavlov

пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:

> Dmitriy,
>
> First of all, thanks for putting this low level details on paper! Current
> and future community members will be grateful to you.
>
> Second, let me be a reviewer of the page and suggest some improvements.
> Until the review is over I would encourage us to hold on with any public
> announcements such as blog post.
>
> I’ve reviewed and edited Ignite Durable Memory section (
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
> <
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>).
> Please address the questions below and I’ll move on with the new sections
> afterwards.
>
> 1) "Different caches and its partitions bring more complex addressing
> scheme but it is still possible to map from page ID to position in
> particular file: pageId => file + offset in this file”.
>
> Probably this appears too early in the doc because above we just
> introduced page ID notion with durable memory and now we’re talking about
> some files? Are these files of ignite persistence? What if the persistence
> is disabled? Honestly, the sentence gets out of the story context.
>
>
> 2) "Also, the tree pages can be merged into one if < 50% of space is used.”
>
> 50% of the whole memory available? Please clarify.
>
>
> 3) "There is one page ID table per memory segment.”
>
> It’s the same segment as it’s described in the main doc, right?
> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
> <
> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
> >
>
>
> 4) "within unsafe segment”
>
> If we want to mention “usafe” here we need to explain what that is.
>
> 5) "If memory amount less than whole index size, Ignite still can operate.”
>
> How do we define “index size”? Is this the page size?
>
>
> —
> Denis
>
>
> > On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
> wrote:
> >
> > Hi Igniters,
> >
> >
> > I am glad to introduce two new wiki articles. These articles contain
> > high-level description of
> >
> > - Ignite Durable Memory (
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > )
> >
> > - and Ignite Persistent Store (
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > )
> >
> > These articles cover basic principles and internal design. Pages may be
> > useful for new contributors during start of working with persistent store
> > code base.
> >
> >
> > I would like to thank all contributors who helped me with my questions.
> > Their answers and support made it possible to start this description.
> >
> >
> > I will appreciate any feedback and questions. Suggestions and edits are
> > welcome, of course.
> >
> >
> > Sincerely,
> >
> > Dmitriy Pavlov
>
>

Dmitriy Pavlov

Re: Ignite Durable Memory & Persistent Store introduction for developers

In reply to this post by dmagda

Hi Denis,

I've updated the section in relation to your questions, please see
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory

I’ve also moved paragraph related to page offset calculation and usage in
files into second article (PDS)

Sincerely,
Dmitriy Pavlov

пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:

Dmitriy Pavlov

Re: Ignite Durable Memory & Persistent Store introduction for developers

Hi Igniters,

I've done several updates in section
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure

These updates releated to optimizations of LoadedPageTable in Ignite 2.5
(Replacement of storage algorithm corresponding to the ID in the page, and
the location of a physical address within the segment algorithm is used
Robin Hood with backward shift. Previous implementation was not resistant
to the ‘page rotation’, a lot of the cells were flooded with Removed state,
and new algorithm does not have ‘removed’ state of cell.)

New section has now picture of segments structure, and describes locking
optimisations in 2.5.

Sincerely,
Dmitriy Pavlov

чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>:

> Hi Denis,
>
> I've updated the section in relation to your questions, please see
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
>
> I’ve also moved paragraph related to page offset calculation and usage in
> files into second article (PDS)
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:
>
>> Dmitriy,
>
>
>>
>> First of all, thanks for putting this low level details on paper! Current
>> and future community members will be grateful to you.
>>
>> Second, let me be a reviewer of the page and suggest some improvements.
>> Until the review is over I would encourage us to hold on with any public
>> announcements such as blog post.
>>
>> I’ve reviewed and edited Ignite Durable Memory section (
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
>> <
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>).
>> Please address the questions below and I’ll move on with the new sections
>> afterwards.
>>
>> 1) "Different caches and its partitions bring more complex addressing
>> scheme but it is still possible to map from page ID to position in
>> particular file: pageId => file + offset in this file”.
>>
>> Probably this appears too early in the doc because above we just
>> introduced page ID notion with durable memory and now we’re talking about
>> some files? Are these files of ignite persistence? What if the persistence
>> is disabled? Honestly, the sentence gets out of the story context.
>>
>>
>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>> used.”
>>
>> 50% of the whole memory available? Please clarify.
>>
>>
>> 3) "There is one page ID table per memory segment.”
>>
>> It’s the same segment as it’s described in the main doc, right?
>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
>> <
>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
>> >
>>
>>
>> 4) "within unsafe segment”
>>
>> If we want to mention “usafe” here we need to explain what that is.
>>
>> 5) "If memory amount less than whole index size, Ignite still can
>> operate.”
>>
>> How do we define “index size”? Is this the page size?
>>
>>
>> —
>> Denis
>>
>>
>> > On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
>> wrote:
>> >
>> > Hi Igniters,
>> >
>> >
>> > I am glad to introduce two new wiki articles. These articles contain
>> > high-level description of
>> >
>> > - Ignite Durable Memory (
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
>> > )
>> >
>> > - and Ignite Persistent Store (
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>> > )
>> >
>> > These articles cover basic principles and internal design. Pages may be
>> > useful for new contributors during start of working with persistent
>> store
>> > code base.
>> >
>> >
>> > I would like to thank all contributors who helped me with my questions.
>> > Their answers and support made it possible to start this description.
>> >
>> >
>> > I will appreciate any feedback and questions. Suggestions and edits are
>> > welcome, of course.
>> >
>> >
>> > Sincerely,
>> >
>> > Dmitriy Pavlov
>>
>>

Ivan Rakov

Re: Ignite Durable Memory & Persistent Store introduction for developers

Igniters,

I've added some info about Pages Write Throttling and WAL compaction to
the "Ignite Persistent Store - under the hood" document.
Please let me know if anything is not clear enough.

Best Regards,
Ivan Rakov

On 26.03.2018 0:18, Dmitry Pavlov wrote:

> Hi Igniters,
>
> I've done several updates in section
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure
>
> These updates releated to optimizations of LoadedPageTable in Ignite 2.5
> (Replacement of storage algorithm corresponding to the ID in the page, and
> the location of a physical address within the segment algorithm is used
> Robin Hood with backward shift. Previous implementation was not resistant
> to the ‘page rotation’, a lot of the cells were flooded with Removed state,
> and new algorithm does not have ‘removed’ state of cell.)
>
> New section has now picture of segments structure, and describes locking
> optimisations in 2.5.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>:
>
>> Hi Denis,
>>
>> I've updated the section in relation to your questions, please see
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
>>
>> I’ve also moved paragraph related to page offset calculation and usage in
>> files into second article (PDS)
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:
>>
>>> Dmitriy,
>>
>>> First of all, thanks for putting this low level details on paper! Current
>>> and future community members will be grateful to you.
>>>
>>> Second, let me be a reviewer of the page and suggest some improvements.
>>> Until the review is over I would encourage us to hold on with any public
>>> announcements such as blog post.
>>>
>>> I’ve reviewed and edited Ignite Durable Memory section (
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
>>> <
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>).
>>> Please address the questions below and I’ll move on with the new sections
>>> afterwards.
>>>
>>> 1) "Different caches and its partitions bring more complex addressing
>>> scheme but it is still possible to map from page ID to position in
>>> particular file: pageId => file + offset in this file”.
>>>
>>> Probably this appears too early in the doc because above we just
>>> introduced page ID notion with durable memory and now we’re talking about
>>> some files? Are these files of ignite persistence? What if the persistence
>>> is disabled? Honestly, the sentence gets out of the story context.
>>>
>>>
>>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>>> used.”
>>>
>>> 50% of the whole memory available? Please clarify.
>>>
>>>
>>> 3) "There is one page ID table per memory segment.”
>>>
>>> It’s the same segment as it’s described in the main doc, right?
>>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
>>> <
>>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
>>>
>>> 4) "within unsafe segment”
>>>
>>> If we want to mention “usafe” here we need to explain what that is.
>>>
>>> 5) "If memory amount less than whole index size, Ignite still can
>>> operate.”
>>>
>>> How do we define “index size”? Is this the page size?
>>>
>>>
>>> —
>>> Denis
>>>
>>>
>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
>>> wrote:
>>>> Hi Igniters,
>>>>
>>>>
>>>> I am glad to introduce two new wiki articles. These articles contain
>>>> high-level description of
>>>>
>>>> - Ignite Durable Memory (
>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
>>>> )
>>>>
>>>> - and Ignite Persistent Store (
>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>>>> )
>>>>
>>>> These articles cover basic principles and internal design. Pages may be
>>>> useful for new contributors during start of working with persistent
>>> store
>>>> code base.
>>>>
>>>>
>>>> I would like to thank all contributors who helped me with my questions.
>>>> Their answers and support made it possible to start this description.
>>>>
>>>>
>>>> I will appreciate any feedback and questions. Suggestions and edits are
>>>> welcome, of course.
>>>>
>>>>
>>>> Sincerely,
>>>>
>>>> Dmitriy Pavlov
>>>

dmagda

Re: Ignite Durable Memory & Persistent Store introduction for developers

Hi Ivan,

Should we update the write-throttling section on readme.io? It makes sense
to do if any recommendations have changed:
https://apacheignite.readme.io/docs/durable-memory-tuning#section-pages-writes-throttling

The compaction technique will be introduced and described on readme:
https://issues.apache.org/jira/browse/IGNITE-8090

--
Denis

On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <[hidden email]> wrote:

> Igniters,
>
> I've added some info about Pages Write Throttling and WAL compaction to
> the "Ignite Persistent Store - under the hood" document.
> Please let me know if anything is not clear enough.
>
> Best Regards,
> Ivan Rakov
>
>
> On 26.03.2018 0:18, Dmitry Pavlov wrote:
>
>> Hi Igniters,
>>
>> I've done several updates in section
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>> thehood-Regionandsegmentstructure
>>
>> These updates releated to optimizations of LoadedPageTable in Ignite 2.5
>> (Replacement of storage algorithm corresponding to the ID in the page, and
>> the location of a physical address within the segment algorithm is used
>> Robin Hood with backward shift. Previous implementation was not resistant
>> to the ‘page rotation’, a lot of the cells were flooded with Removed
>> state,
>> and new algorithm does not have ‘removed’ state of cell.)
>>
>> New section has now picture of segments structure, and describes locking
>> optimisations in 2.5.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>:
>>
>> Hi Denis,
>>>
>>> I've updated the section in relation to your questions, please see
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>> thehood-IgniteDurableMemory
>>>
>>> I’ve also moved paragraph related to page offset calculation and usage in
>>> files into second article (PDS)
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:
>>>
>>> Dmitriy,
>>>>
>>>
>>> First of all, thanks for putting this low level details on paper! Current
>>>> and future community members will be grateful to you.
>>>>
>>>> Second, let me be a reviewer of the page and suggest some improvements.
>>>> Until the review is over I would encourage us to hold on with any public
>>>> announcements such as blog post.
>>>>
>>>> I’ve reviewed and edited Ignite Durable Memory section (
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>> thehood-IgniteDurableMemory
>>>> <
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>> thehood-IgniteDurableMemory>).
>>>> Please address the questions below and I’ll move on with the new
>>>> sections
>>>> afterwards.
>>>>
>>>> 1) "Different caches and its partitions bring more complex addressing
>>>> scheme but it is still possible to map from page ID to position in
>>>> particular file: pageId => file + offset in this file”.
>>>>
>>>> Probably this appears too early in the doc because above we just
>>>> introduced page ID notion with durable memory and now we’re talking
>>>> about
>>>> some files? Are these files of ignite persistence? What if the
>>>> persistence
>>>> is disabled? Honestly, the sentence gets out of the story context.
>>>>
>>>>
>>>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>>>> used.”
>>>>
>>>> 50% of the whole memory available? Please clarify.
>>>>
>>>>
>>>> 3) "There is one page ID table per memory segment.”
>>>>
>>>> It’s the same segment as it’s described in the main doc, right?
>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>> ion-memory-segment
>>>> <
>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>> ion-memory-segment
>>>>
>>>> 4) "within unsafe segment”
>>>>
>>>> If we want to mention “usafe” here we need to explain what that is.
>>>>
>>>> 5) "If memory amount less than whole index size, Ignite still can
>>>> operate.”
>>>>
>>>> How do we define “index size”? Is this the page size?
>>>>
>>>>
>>>> —
>>>> Denis
>>>>
>>>>
>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
>>>>>
>>>> wrote:
>>>>
>>>>> Hi Igniters,
>>>>>
>>>>>
>>>>> I am glad to introduce two new wiki articles. These articles contain
>>>>> high-level description of
>>>>>
>>>>> - Ignite Durable Memory (
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood
>>>>
>>>>> )
>>>>>
>>>>> - and Ignite Persistent Store (
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Persistent+Store+-+under+the+hood
>>>>
>>>>> )
>>>>>
>>>>> These articles cover basic principles and internal design. Pages may be
>>>>> useful for new contributors during start of working with persistent
>>>>>
>>>> store
>>>>
>>>>> code base.
>>>>>
>>>>>
>>>>> I would like to thank all contributors who helped me with my questions.
>>>>> Their answers and support made it possible to start this description.
>>>>>
>>>>>
>>>>> I will appreciate any feedback and questions. Suggestions and edits are
>>>>> welcome, of course.
>>>>>
>>>>>
>>>>> Sincerely,
>>>>>
>>>>> Dmitriy Pavlov
>>>>>
>>>>
>>>>
>

Dmitriy Pavlov

Re: Ignite Durable Memory & Persistent Store introduction for developers

In reply to this post by Ivan Rakov

Hi Ivan,

Excellent addition to the article. Thank you very much for contribution.

I hope Igniters, especially Native Persistence experts will follow your
lead.

Sincerely,
Dmitriy Pavlov

пт, 30 мар. 2018 г. в 11:38, Ivan Rakov <[hidden email]>:

> Igniters,
>
> I've added some info about Pages Write Throttling and WAL compaction to
> the "Ignite Persistent Store - under the hood" document.
> Please let me know if anything is not clear enough.
>
> Best Regards,
> Ivan Rakov
>
> On 26.03.2018 0:18, Dmitry Pavlov wrote:
> > Hi Igniters,
> >
> > I've done several updates in section
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure
> >
> > These updates releated to optimizations of LoadedPageTable in Ignite 2.5
> > (Replacement of storage algorithm corresponding to the ID in the page,
> and
> > the location of a physical address within the segment algorithm is used
> > Robin Hood with backward shift. Previous implementation was not resistant
> > to the ‘page rotation’, a lot of the cells were flooded with Removed
> state,
> > and new algorithm does not have ‘removed’ state of cell.)
> >
> > New section has now picture of segments structure, and describes locking
> > optimisations in 2.5.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>:
> >
> >> Hi Denis,
> >>
> >> I've updated the section in relation to your questions, please see
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
> >>
> >> I’ve also moved paragraph related to page offset calculation and usage
> in
> >> files into second article (PDS)
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:
> >>
> >>> Dmitriy,
> >>
> >>> First of all, thanks for putting this low level details on paper!
> Current
> >>> and future community members will be grateful to you.
> >>>
> >>> Second, let me be a reviewer of the page and suggest some improvements.
> >>> Until the review is over I would encourage us to hold on with any
> public
> >>> announcements such as blog post.
> >>>
> >>> I’ve reviewed and edited Ignite Durable Memory section (
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
> >>> <
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
> >).
> >>> Please address the questions below and I’ll move on with the new
> sections
> >>> afterwards.
> >>>
> >>> 1) "Different caches and its partitions bring more complex addressing
> >>> scheme but it is still possible to map from page ID to position in
> >>> particular file: pageId => file + offset in this file”.
> >>>
> >>> Probably this appears too early in the doc because above we just
> >>> introduced page ID notion with durable memory and now we’re talking
> about
> >>> some files? Are these files of ignite persistence? What if the
> persistence
> >>> is disabled? Honestly, the sentence gets out of the story context.
> >>>
> >>>
> >>> 2) "Also, the tree pages can be merged into one if < 50% of space is
> >>> used.”
> >>>
> >>> 50% of the whole memory available? Please clarify.
> >>>
> >>>
> >>> 3) "There is one page ID table per memory segment.”
> >>>
> >>> It’s the same segment as it’s described in the main doc, right?
> >>>
> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
> >>> <
> >>>
> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
> >>>
> >>> 4) "within unsafe segment”
> >>>
> >>> If we want to mention “usafe” here we need to explain what that is.
> >>>
> >>> 5) "If memory amount less than whole index size, Ignite still can
> >>> operate.”
> >>>
> >>> How do we define “index size”? Is this the page size?
> >>>
> >>>
> >>> —
> >>> Denis
> >>>
> >>>
> >>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
> >>> wrote:
> >>>> Hi Igniters,
> >>>>
> >>>>
> >>>> I am glad to introduce two new wiki articles. These articles contain
> >>>> high-level description of
> >>>>
> >>>> - Ignite Durable Memory (
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> >>>> )
> >>>>
> >>>> - and Ignite Persistent Store (
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> >>>> )
> >>>>
> >>>> These articles cover basic principles and internal design. Pages may
> be
> >>>> useful for new contributors during start of working with persistent
> >>> store
> >>>> code base.
> >>>>
> >>>>
> >>>> I would like to thank all contributors who helped me with my
> questions.
> >>>> Their answers and support made it possible to start this description.
> >>>>
> >>>>
> >>>> I will appreciate any feedback and questions. Suggestions and edits
> are
> >>>> welcome, of course.
> >>>>
> >>>>
> >>>> Sincerely,
> >>>>
> >>>> Dmitriy Pavlov
> >>>
>
>

Ivan Rakov

Re: Ignite Durable Memory & Persistent Store introduction for developers

In reply to this post by dmagda

Denis, thanks for pointing this out.

I think that section on readme.io is good enough.
I'd only recommend to highlight "write throttling algorithm" words with
link to "Pages Write Throttling" section of "Persistence under the
hood". What's already described on readme.io is enough for "Durable
Memory Tuning" page, but if advanced user wants to know details about
throttling, he'll be navigated to page with internals.

Best Regards,
Ivan Rakov

On 30.03.2018 19:48, Denis Magda wrote:

> Hi Ivan,
>
> Should we update the write-throttling section on readme.io? It makes sense
> to do if any recommendations have changed:
> https://apacheignite.readme.io/docs/durable-memory-tuning#section-pages-writes-throttling
>
> The compaction technique will be introduced and described on readme:
> https://issues.apache.org/jira/browse/IGNITE-8090
>
> --
> Denis
>
> On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <[hidden email]> wrote:
>
>> Igniters,
>>
>> I've added some info about Pages Write Throttling and WAL compaction to
>> the "Ignite Persistent Store - under the hood" document.
>> Please let me know if anything is not clear enough.
>>
>> Best Regards,
>> Ivan Rakov
>>
>>
>> On 26.03.2018 0:18, Dmitry Pavlov wrote:
>>
>>> Hi Igniters,
>>>
>>> I've done several updates in section
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>> thehood-Regionandsegmentstructure
>>>
>>> These updates releated to optimizations of LoadedPageTable in Ignite 2.5
>>> (Replacement of storage algorithm corresponding to the ID in the page, and
>>> the location of a physical address within the segment algorithm is used
>>> Robin Hood with backward shift. Previous implementation was not resistant
>>> to the ‘page rotation’, a lot of the cells were flooded with Removed
>>> state,
>>> and new algorithm does not have ‘removed’ state of cell.)
>>>
>>> New section has now picture of segments structure, and describes locking
>>> optimisations in 2.5.
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>:
>>>
>>> Hi Denis,
>>>> I've updated the section in relation to your questions, please see
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>> thehood-IgniteDurableMemory
>>>>
>>>> I’ve also moved paragraph related to page offset calculation and usage in
>>>> files into second article (PDS)
>>>>
>>>> Sincerely,
>>>> Dmitriy Pavlov
>>>>
>>>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:
>>>>
>>>> Dmitriy,
>>>> First of all, thanks for putting this low level details on paper! Current
>>>>> and future community members will be grateful to you.
>>>>>
>>>>> Second, let me be a reviewer of the page and suggest some improvements.
>>>>> Until the review is over I would encourage us to hold on with any public
>>>>> announcements such as blog post.
>>>>>
>>>>> I’ve reviewed and edited Ignite Durable Memory section (
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>>> thehood-IgniteDurableMemory
>>>>> <
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>>> thehood-IgniteDurableMemory>).
>>>>> Please address the questions below and I’ll move on with the new
>>>>> sections
>>>>> afterwards.
>>>>>
>>>>> 1) "Different caches and its partitions bring more complex addressing
>>>>> scheme but it is still possible to map from page ID to position in
>>>>> particular file: pageId => file + offset in this file”.
>>>>>
>>>>> Probably this appears too early in the doc because above we just
>>>>> introduced page ID notion with durable memory and now we’re talking
>>>>> about
>>>>> some files? Are these files of ignite persistence? What if the
>>>>> persistence
>>>>> is disabled? Honestly, the sentence gets out of the story context.
>>>>>
>>>>>
>>>>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>>>>> used.”
>>>>>
>>>>> 50% of the whole memory available? Please clarify.
>>>>>
>>>>>
>>>>> 3) "There is one page ID table per memory segment.”
>>>>>
>>>>> It’s the same segment as it’s described in the main doc, right?
>>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>>> ion-memory-segment
>>>>> <
>>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>>> ion-memory-segment
>>>>>
>>>>> 4) "within unsafe segment”
>>>>>
>>>>> If we want to mention “usafe” here we need to explain what that is.
>>>>>
>>>>> 5) "If memory amount less than whole index size, Ignite still can
>>>>> operate.”
>>>>>
>>>>> How do we define “index size”? Is this the page size?
>>>>>
>>>>>
>>>>> —
>>>>> Denis
>>>>>
>>>>>
>>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Hi Igniters,
>>>>>>
>>>>>>
>>>>>> I am glad to introduce two new wiki articles. These articles contain
>>>>>> high-level description of
>>>>>>
>>>>>> - Ignite Durable Memory (
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>> Durable+Memory+-+under+the+hood
>>>>>
>>>>>> )
>>>>>>
>>>>>> - and Ignite Persistent Store (
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>> Persistent+Store+-+under+the+hood
>>>>>
>>>>>> )
>>>>>>
>>>>>> These articles cover basic principles and internal design. Pages may be
>>>>>> useful for new contributors during start of working with persistent
>>>>>>
>>>>> store
>>>>>
>>>>>> code base.
>>>>>>
>>>>>>
>>>>>> I would like to thank all contributors who helped me with my questions.
>>>>>> Their answers and support made it possible to start this description.
>>>>>>
>>>>>>
>>>>>> I will appreciate any feedback and questions. Suggestions and edits are
>>>>>> welcome, of course.
>>>>>>
>>>>>>
>>>>>> Sincerely,
>>>>>>
>>>>>> Dmitriy Pavlov
>>>>>>
>>>>>

dmagda

Re: Ignite Durable Memory & Persistent Store introduction for developers

Ivan,

Sure, not a problem. Did more, created a hard-not-to-notice block
titled Pages Write Throttling in a Nutshell.

Thanks!
Denis

On Fri, Mar 30, 2018 at 10:00 AM, Ivan Rakov <[hidden email]> wrote:

> Denis, thanks for pointing this out.
>
> I think that section on readme.io is good enough.
> I'd only recommend to highlight "write throttling algorithm" words with
> link to "Pages Write Throttling" section of "Persistence under the hood".
> What's already described on readme.io is enough for "Durable Memory
> Tuning" page, but if advanced user wants to know details about throttling,
> he'll be navigated to page with internals.
>
> Best Regards,
> Ivan Rakov
>
>
> On 30.03.2018 19:48, Denis Magda wrote:
>
>> Hi Ivan,
>>
>> Should we update the write-throttling section on readme.io? It makes
>> sense
>> to do if any recommendations have changed:
>> https://apacheignite.readme.io/docs/durable-memory-tuning#se
>> ction-pages-writes-throttling
>>
>> The compaction technique will be introduced and described on readme:
>> https://issues.apache.org/jira/browse/IGNITE-8090
>>
>> --
>> Denis
>>
>> On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <[hidden email]>
>> wrote:
>>
>> Igniters,
>>>
>>> I've added some info about Pages Write Throttling and WAL compaction to
>>> the "Ignite Persistent Store - under the hood" document.
>>> Please let me know if anything is not clear enough.
>>>
>>> Best Regards,
>>> Ivan Rakov
>>>
>>>
>>> On 26.03.2018 0:18, Dmitry Pavlov wrote:
>>>
>>> Hi Igniters,
>>>>
>>>> I've done several updates in section
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>> thehood-Regionandsegmentstructure
>>>>
>>>> These updates releated to optimizations of LoadedPageTable in Ignite 2.5
>>>> (Replacement of storage algorithm corresponding to the ID in the page,
>>>> and
>>>> the location of a physical address within the segment algorithm is used
>>>> Robin Hood with backward shift. Previous implementation was not
>>>> resistant
>>>> to the ‘page rotation’, a lot of the cells were flooded with Removed
>>>> state,
>>>> and new algorithm does not have ‘removed’ state of cell.)
>>>>
>>>> New section has now picture of segments structure, and describes locking
>>>> optimisations in 2.5.
>>>>
>>>> Sincerely,
>>>> Dmitriy Pavlov
>>>>
>>>> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>:
>>>>
>>>> Hi Denis,
>>>>
>>>>> I've updated the section in relation to your questions, please see
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>>> thehood-IgniteDurableMemory
>>>>>
>>>>> I’ve also moved paragraph related to page offset calculation and usage
>>>>> in
>>>>> files into second article (PDS)
>>>>>
>>>>> Sincerely,
>>>>> Dmitriy Pavlov
>>>>>
>>>>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>:
>>>>>
>>>>> Dmitriy,
>>>>> First of all, thanks for putting this low level details on paper!
>>>>> Current
>>>>>
>>>>>> and future community members will be grateful to you.
>>>>>>
>>>>>> Second, let me be a reviewer of the page and suggest some
>>>>>> improvements.
>>>>>> Until the review is over I would encourage us to hold on with any
>>>>>> public
>>>>>> announcements such as blog post.
>>>>>>
>>>>>> I’ve reviewed and edited Ignite Durable Memory section (
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>>>> thehood-IgniteDurableMemory
>>>>>> <
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>>>> thehood-IgniteDurableMemory>).
>>>>>> Please address the questions below and I’ll move on with the new
>>>>>> sections
>>>>>> afterwards.
>>>>>>
>>>>>> 1) "Different caches and its partitions bring more complex addressing
>>>>>> scheme but it is still possible to map from page ID to position in
>>>>>> particular file: pageId => file + offset in this file”.
>>>>>>
>>>>>> Probably this appears too early in the doc because above we just
>>>>>> introduced page ID notion with durable memory and now we’re talking
>>>>>> about
>>>>>> some files? Are these files of ignite persistence? What if the
>>>>>> persistence
>>>>>> is disabled? Honestly, the sentence gets out of the story context.
>>>>>>
>>>>>>
>>>>>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>>>>>> used.”
>>>>>>
>>>>>> 50% of the whole memory available? Please clarify.
>>>>>>
>>>>>>
>>>>>> 3) "There is one page ID table per memory segment.”
>>>>>>
>>>>>> It’s the same segment as it’s described in the main doc, right?
>>>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>>>> ion-memory-segment
>>>>>> <
>>>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>>>> ion-memory-segment
>>>>>>
>>>>>> 4) "within unsafe segment”
>>>>>>
>>>>>> If we want to mention “usafe” here we need to explain what that is.
>>>>>>
>>>>>> 5) "If memory amount less than whole index size, Ignite still can
>>>>>> operate.”
>>>>>>
>>>>>> How do we define “index size”? Is this the page size?
>>>>>>
>>>>>>
>>>>>> —
>>>>>> Denis
>>>>>>
>>>>>>
>>>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Igniters,
>>>>>>>
>>>>>>>
>>>>>>> I am glad to introduce two new wiki articles. These articles contain
>>>>>>> high-level description of
>>>>>>>
>>>>>>> - Ignite Durable Memory (
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>>>>
>>>>>> Durable+Memory+-+under+the+hood
>>>>>>
>>>>>> )
>>>>>>>
>>>>>>> - and Ignite Persistent Store (
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>>>>>
>>>>>> Persistent+Store+-+under+the+hood
>>>>>>
>>>>>> )
>>>>>>>
>>>>>>> These articles cover basic principles and internal design. Pages may
>>>>>>> be
>>>>>>> useful for new contributors during start of working with persistent
>>>>>>>
>>>>>>> store
>>>>>>
>>>>>> code base.
>>>>>>>
>>>>>>>
>>>>>>> I would like to thank all contributors who helped me with my
>>>>>>> questions.
>>>>>>> Their answers and support made it possible to start this description.
>>>>>>>
>>>>>>>
>>>>>>> I will appreciate any feedback and questions. Suggestions and edits
>>>>>>> are
>>>>>>> welcome, of course.
>>>>>>>
>>>>>>>
>>>>>>> Sincerely,
>>>>>>>
>>>>>>> Dmitriy Pavlov
>>>>>>>
>>>>>>>
>>>>>>
>