Hi Igniters,
I am glad to introduce two new wiki articles. These articles contain high-level description of - Ignite Durable Memory ( https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood ) - and Ignite Persistent Store ( https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood ) These articles cover basic principles and internal design. Pages may be useful for new contributors during start of working with persistent store code base. I would like to thank all contributors who helped me with my questions. Their answers and support made it possible to start this description. I will appreciate any feedback and questions. Suggestions and edits are welcome, of course. Sincerely, Dmitriy Pavlov |
Awesome!
Thanks! On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]> wrote: > Hi Igniters, > > > I am glad to introduce two new wiki articles. These articles contain > high-level description of > > - Ignite Durable Memory ( > https://cwiki.apache.org/confluence/display/IGNITE/ > Ignite+Durable+Memory+-+under+the+hood > ) > > - and Ignite Persistent Store ( > https://cwiki.apache.org/confluence/display/IGNITE/ > Ignite+Persistent+Store+-+under+the+hood > ) > > These articles cover basic principles and internal design. Pages may be > useful for new contributors during start of working with persistent store > code base. > > > I would like to thank all contributors who helped me with my questions. > Their answers and support made it possible to start this description. > > > I will appreciate any feedback and questions. Suggestions and edits are > welcome, of course. > > > Sincerely, > > Dmitriy Pavlov > |
In reply to this post by Dmitriy Pavlov
Dmitriy, this is very useful! Any chance you can also blog about it?
On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]> wrote: > Hi Igniters, > > > I am glad to introduce two new wiki articles. These articles contain > high-level description of > > - Ignite Durable Memory ( > https://cwiki.apache.org/confluence/display/IGNITE/ > Ignite+Durable+Memory+-+under+the+hood > ) > > - and Ignite Persistent Store ( > https://cwiki.apache.org/confluence/display/IGNITE/ > Ignite+Persistent+Store+-+under+the+hood > ) > > These articles cover basic principles and internal design. Pages may be > useful for new contributors during start of working with persistent store > code base. > > > I would like to thank all contributors who helped me with my questions. > Their answers and support made it possible to start this description. > > > I will appreciate any feedback and questions. Suggestions and edits are > welcome, of course. > > > Sincerely, > > Dmitriy Pavlov > |
Sure, I can blog about it, but timelines depend on free time. Which
language is preferred? Russian or English? Also I plan to extend existing wiki pages with more details, and add new pages explaining Ignite to newbies. This activity depends on interest of community and new questions related to pages. чт, 31 авг. 2017 г. в 3:53, Dmitriy Setrakyan <[hidden email]>: > Dmitriy, this is very useful! Any chance you can also blog about it? > > On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]> > wrote: > > > Hi Igniters, > > > > > > I am glad to introduce two new wiki articles. These articles contain > > high-level description of > > > > - Ignite Durable Memory ( > > https://cwiki.apache.org/confluence/display/IGNITE/ > > Ignite+Durable+Memory+-+under+the+hood > > ) > > > > - and Ignite Persistent Store ( > > https://cwiki.apache.org/confluence/display/IGNITE/ > > Ignite+Persistent+Store+-+under+the+hood > > ) > > > > These articles cover basic principles and internal design. Pages may be > > useful for new contributors during start of working with persistent store > > code base. > > > > > > I would like to thank all contributors who helped me with my questions. > > Their answers and support made it possible to start this description. > > > > > > I will appreciate any feedback and questions. Suggestions and edits are > > welcome, of course. > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > |
Dmitry,
Both languages are preferable. :) English has priority. 31 авг. 2017 г. 1:06 ПП пользователь "Dmitry Pavlov" <[hidden email]> написал: > Sure, I can blog about it, but timelines depend on free time. Which > language is preferred? Russian or English? Also I plan to extend existing > wiki pages with more details, and add new pages explaining Ignite to > newbies. This activity depends on interest of community and new questions > related to pages. > > чт, 31 авг. 2017 г. в 3:53, Dmitriy Setrakyan <[hidden email]>: > > > Dmitriy, this is very useful! Any chance you can also blog about it? > > > > On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]> > > wrote: > > > > > Hi Igniters, > > > > > > > > > I am glad to introduce two new wiki articles. These articles contain > > > high-level description of > > > > > > - Ignite Durable Memory ( > > > https://cwiki.apache.org/confluence/display/IGNITE/ > > > Ignite+Durable+Memory+-+under+the+hood > > > ) > > > > > > - and Ignite Persistent Store ( > > > https://cwiki.apache.org/confluence/display/IGNITE/ > > > Ignite+Persistent+Store+-+under+the+hood > > > ) > > > > > > These articles cover basic principles and internal design. Pages may be > > > useful for new contributors during start of working with persistent > store > > > code base. > > > > > > > > > I would like to thank all contributors who helped me with my questions. > > > Their answers and support made it possible to start this description. > > > > > > > > > I will appreciate any feedback and questions. Suggestions and edits are > > > welcome, of course. > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > |
I will help to translate into chinese.
在 2017/8/31 下午6:39, Alexey Dmitriev 写道: > Dmitry, > Both languages are preferable. :) > English has priority. > > 31 авг. 2017 г. 1:06 ПП пользователь "Dmitry Pavlov" <[hidden email]> > написал: > >> Sure, I can blog about it, but timelines depend on free time. Which >> language is preferred? Russian or English? Also I plan to extend existing >> wiki pages with more details, and add new pages explaining Ignite to >> newbies. This activity depends on interest of community and new questions >> related to pages. >> >> чт, 31 авг. 2017 г. в 3:53, Dmitriy Setrakyan <[hidden email]>: >> >>> Dmitriy, this is very useful! Any chance you can also blog about it? >>> >>> On Wed, Aug 30, 2017 at 6:24 PM, Dmitry Pavlov <[hidden email]> >>> wrote: >>> >>>> Hi Igniters, >>>> >>>> >>>> I am glad to introduce two new wiki articles. These articles contain >>>> high-level description of >>>> >>>> - Ignite Durable Memory ( >>>> https://cwiki.apache.org/confluence/display/IGNITE/ >>>> Ignite+Durable+Memory+-+under+the+hood >>>> ) >>>> >>>> - and Ignite Persistent Store ( >>>> https://cwiki.apache.org/confluence/display/IGNITE/ >>>> Ignite+Persistent+Store+-+under+the+hood >>>> ) >>>> >>>> These articles cover basic principles and internal design. Pages may be >>>> useful for new contributors during start of working with persistent >> store >>>> code base. >>>> >>>> >>>> I would like to thank all contributors who helped me with my questions. >>>> Their answers and support made it possible to start this description. >>>> >>>> >>>> I will appreciate any feedback and questions. Suggestions and edits are >>>> welcome, of course. >>>> >>>> >>>> Sincerely, >>>> >>>> Dmitriy Pavlov >>>> |
In reply to this post by Dmitriy Pavlov
Dmitriy,
First of all, thanks for putting this low level details on paper! Current and future community members will be grateful to you. Second, let me be a reviewer of the page and suggest some improvements. Until the review is over I would encourage us to hold on with any public announcements such as blog post. I’ve reviewed and edited Ignite Durable Memory section (https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory <https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>). Please address the questions below and I’ll move on with the new sections afterwards. 1) "Different caches and its partitions bring more complex addressing scheme but it is still possible to map from page ID to position in particular file: pageId => file + offset in this file”. Probably this appears too early in the doc because above we just introduced page ID notion with durable memory and now we’re talking about some files? Are these files of ignite persistence? What if the persistence is disabled? Honestly, the sentence gets out of the story context. 2) "Also, the tree pages can be merged into one if < 50% of space is used.” 50% of the whole memory available? Please clarify. 3) "There is one page ID table per memory segment.” It’s the same segment as it’s described in the main doc, right? https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment <https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment> 4) "within unsafe segment” If we want to mention “usafe” here we need to explain what that is. 5) "If memory amount less than whole index size, Ignite still can operate.” How do we define “index size”? Is this the page size? — Denis > On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> wrote: > > Hi Igniters, > > > I am glad to introduce two new wiki articles. These articles contain > high-level description of > > - Ignite Durable Memory ( > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood > ) > > - and Ignite Persistent Store ( > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood > ) > > These articles cover basic principles and internal design. Pages may be > useful for new contributors during start of working with persistent store > code base. > > > I would like to thank all contributors who helped me with my questions. > Their answers and support made it possible to start this description. > > > I will appreciate any feedback and questions. Suggestions and edits are > welcome, of course. > > > Sincerely, > > Dmitriy Pavlov |
Hi Denis,
Sure, I would be glad if you have some time to review the page. Thank you for edits and careful review! According to recent questions and feedback from Igniters, I would also like to extend section about memory organization. Please give me a couple of days to address your questions and extend wiki. Sincerely, Dmitriy Pavlov пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: > Dmitriy, > > First of all, thanks for putting this low level details on paper! Current > and future community members will be grateful to you. > > Second, let me be a reviewer of the page and suggest some improvements. > Until the review is over I would encourage us to hold on with any public > announcements such as blog post. > > I’ve reviewed and edited Ignite Durable Memory section ( > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory > < > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>). > Please address the questions below and I’ll move on with the new sections > afterwards. > > 1) "Different caches and its partitions bring more complex addressing > scheme but it is still possible to map from page ID to position in > particular file: pageId => file + offset in this file”. > > Probably this appears too early in the doc because above we just > introduced page ID notion with durable memory and now we’re talking about > some files? Are these files of ignite persistence? What if the persistence > is disabled? Honestly, the sentence gets out of the story context. > > > 2) "Also, the tree pages can be merged into one if < 50% of space is used.” > > 50% of the whole memory available? Please clarify. > > > 3) "There is one page ID table per memory segment.” > > It’s the same segment as it’s described in the main doc, right? > https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment > < > https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment > > > > > 4) "within unsafe segment” > > If we want to mention “usafe” here we need to explain what that is. > > 5) "If memory amount less than whole index size, Ignite still can operate.” > > How do we define “index size”? Is this the page size? > > > — > Denis > > > > On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> > wrote: > > > > Hi Igniters, > > > > > > I am glad to introduce two new wiki articles. These articles contain > > high-level description of > > > > - Ignite Durable Memory ( > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood > > ) > > > > - and Ignite Persistent Store ( > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood > > ) > > > > These articles cover basic principles and internal design. Pages may be > > useful for new contributors during start of working with persistent store > > code base. > > > > > > I would like to thank all contributors who helped me with my questions. > > Their answers and support made it possible to start this description. > > > > > > I will appreciate any feedback and questions. Suggestions and edits are > > welcome, of course. > > > > > > Sincerely, > > > > Dmitriy Pavlov > > |
In reply to this post by dmagda
Hi Denis,
I've updated the section in relation to your questions, please see https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory I’ve also moved paragraph related to page offset calculation and usage in files into second article (PDS) Sincerely, Dmitriy Pavlov пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: > Dmitriy, > > First of all, thanks for putting this low level details on paper! Current > and future community members will be grateful to you. > > Second, let me be a reviewer of the page and suggest some improvements. > Until the review is over I would encourage us to hold on with any public > announcements such as blog post. > > I’ve reviewed and edited Ignite Durable Memory section ( > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory > < > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>). > Please address the questions below and I’ll move on with the new sections > afterwards. > > 1) "Different caches and its partitions bring more complex addressing > scheme but it is still possible to map from page ID to position in > particular file: pageId => file + offset in this file”. > > Probably this appears too early in the doc because above we just > introduced page ID notion with durable memory and now we’re talking about > some files? Are these files of ignite persistence? What if the persistence > is disabled? Honestly, the sentence gets out of the story context. > > > 2) "Also, the tree pages can be merged into one if < 50% of space is used.” > > 50% of the whole memory available? Please clarify. > > > 3) "There is one page ID table per memory segment.” > > It’s the same segment as it’s described in the main doc, right? > https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment > < > https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment > > > > > 4) "within unsafe segment” > > If we want to mention “usafe” here we need to explain what that is. > > 5) "If memory amount less than whole index size, Ignite still can operate.” > > How do we define “index size”? Is this the page size? > > > — > Denis > > > > On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> > wrote: > > > > Hi Igniters, > > > > > > I am glad to introduce two new wiki articles. These articles contain > > high-level description of > > > > - Ignite Durable Memory ( > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood > > ) > > > > - and Ignite Persistent Store ( > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood > > ) > > > > These articles cover basic principles and internal design. Pages may be > > useful for new contributors during start of working with persistent store > > code base. > > > > > > I would like to thank all contributors who helped me with my questions. > > Their answers and support made it possible to start this description. > > > > > > I will appreciate any feedback and questions. Suggestions and edits are > > welcome, of course. > > > > > > Sincerely, > > > > Dmitriy Pavlov > > |
Hi Igniters,
I've done several updates in section https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure These updates releated to optimizations of LoadedPageTable in Ignite 2.5 (Replacement of storage algorithm corresponding to the ID in the page, and the location of a physical address within the segment algorithm is used Robin Hood with backward shift. Previous implementation was not resistant to the ‘page rotation’, a lot of the cells were flooded with Removed state, and new algorithm does not have ‘removed’ state of cell.) New section has now picture of segments structure, and describes locking optimisations in 2.5. Sincerely, Dmitriy Pavlov чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>: > Hi Denis, > > I've updated the section in relation to your questions, please see > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory > > I’ve also moved paragraph related to page offset calculation and usage in > files into second article (PDS) > > Sincerely, > Dmitriy Pavlov > > пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: > >> Dmitriy, > > >> >> First of all, thanks for putting this low level details on paper! Current >> and future community members will be grateful to you. >> >> Second, let me be a reviewer of the page and suggest some improvements. >> Until the review is over I would encourage us to hold on with any public >> announcements such as blog post. >> >> I’ve reviewed and edited Ignite Durable Memory section ( >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory >> < >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>). >> Please address the questions below and I’ll move on with the new sections >> afterwards. >> >> 1) "Different caches and its partitions bring more complex addressing >> scheme but it is still possible to map from page ID to position in >> particular file: pageId => file + offset in this file”. >> >> Probably this appears too early in the doc because above we just >> introduced page ID notion with durable memory and now we’re talking about >> some files? Are these files of ignite persistence? What if the persistence >> is disabled? Honestly, the sentence gets out of the story context. >> >> >> 2) "Also, the tree pages can be merged into one if < 50% of space is >> used.” >> >> 50% of the whole memory available? Please clarify. >> >> >> 3) "There is one page ID table per memory segment.” >> >> It’s the same segment as it’s described in the main doc, right? >> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment >> < >> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment >> > >> >> >> 4) "within unsafe segment” >> >> If we want to mention “usafe” here we need to explain what that is. >> >> 5) "If memory amount less than whole index size, Ignite still can >> operate.” >> >> How do we define “index size”? Is this the page size? >> >> >> — >> Denis >> >> >> > On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> >> wrote: >> > >> > Hi Igniters, >> > >> > >> > I am glad to introduce two new wiki articles. These articles contain >> > high-level description of >> > >> > - Ignite Durable Memory ( >> > >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood >> > ) >> > >> > - and Ignite Persistent Store ( >> > >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood >> > ) >> > >> > These articles cover basic principles and internal design. Pages may be >> > useful for new contributors during start of working with persistent >> store >> > code base. >> > >> > >> > I would like to thank all contributors who helped me with my questions. >> > Their answers and support made it possible to start this description. >> > >> > >> > I will appreciate any feedback and questions. Suggestions and edits are >> > welcome, of course. >> > >> > >> > Sincerely, >> > >> > Dmitriy Pavlov >> >> |
Igniters,
I've added some info about Pages Write Throttling and WAL compaction to the "Ignite Persistent Store - under the hood" document. Please let me know if anything is not clear enough. Best Regards, Ivan Rakov On 26.03.2018 0:18, Dmitry Pavlov wrote: > Hi Igniters, > > I've done several updates in section > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure > > These updates releated to optimizations of LoadedPageTable in Ignite 2.5 > (Replacement of storage algorithm corresponding to the ID in the page, and > the location of a physical address within the segment algorithm is used > Robin Hood with backward shift. Previous implementation was not resistant > to the ‘page rotation’, a lot of the cells were flooded with Removed state, > and new algorithm does not have ‘removed’ state of cell.) > > New section has now picture of segments structure, and describes locking > optimisations in 2.5. > > Sincerely, > Dmitriy Pavlov > > чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>: > >> Hi Denis, >> >> I've updated the section in relation to your questions, please see >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory >> >> I’ve also moved paragraph related to page offset calculation and usage in >> files into second article (PDS) >> >> Sincerely, >> Dmitriy Pavlov >> >> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: >> >>> Dmitriy, >> >>> First of all, thanks for putting this low level details on paper! Current >>> and future community members will be grateful to you. >>> >>> Second, let me be a reviewer of the page and suggest some improvements. >>> Until the review is over I would encourage us to hold on with any public >>> announcements such as blog post. >>> >>> I’ve reviewed and edited Ignite Durable Memory section ( >>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory >>> < >>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>). >>> Please address the questions below and I’ll move on with the new sections >>> afterwards. >>> >>> 1) "Different caches and its partitions bring more complex addressing >>> scheme but it is still possible to map from page ID to position in >>> particular file: pageId => file + offset in this file”. >>> >>> Probably this appears too early in the doc because above we just >>> introduced page ID notion with durable memory and now we’re talking about >>> some files? Are these files of ignite persistence? What if the persistence >>> is disabled? Honestly, the sentence gets out of the story context. >>> >>> >>> 2) "Also, the tree pages can be merged into one if < 50% of space is >>> used.” >>> >>> 50% of the whole memory available? Please clarify. >>> >>> >>> 3) "There is one page ID table per memory segment.” >>> >>> It’s the same segment as it’s described in the main doc, right? >>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment >>> < >>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment >>> >>> 4) "within unsafe segment” >>> >>> If we want to mention “usafe” here we need to explain what that is. >>> >>> 5) "If memory amount less than whole index size, Ignite still can >>> operate.” >>> >>> How do we define “index size”? Is this the page size? >>> >>> >>> — >>> Denis >>> >>> >>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> >>> wrote: >>>> Hi Igniters, >>>> >>>> >>>> I am glad to introduce two new wiki articles. These articles contain >>>> high-level description of >>>> >>>> - Ignite Durable Memory ( >>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood >>>> ) >>>> >>>> - and Ignite Persistent Store ( >>>> >>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood >>>> ) >>>> >>>> These articles cover basic principles and internal design. Pages may be >>>> useful for new contributors during start of working with persistent >>> store >>>> code base. >>>> >>>> >>>> I would like to thank all contributors who helped me with my questions. >>>> Their answers and support made it possible to start this description. >>>> >>>> >>>> I will appreciate any feedback and questions. Suggestions and edits are >>>> welcome, of course. >>>> >>>> >>>> Sincerely, >>>> >>>> Dmitriy Pavlov >>> |
Hi Ivan,
Should we update the write-throttling section on readme.io? It makes sense to do if any recommendations have changed: https://apacheignite.readme.io/docs/durable-memory-tuning#section-pages-writes-throttling The compaction technique will be introduced and described on readme: https://issues.apache.org/jira/browse/IGNITE-8090 -- Denis On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <[hidden email]> wrote: > Igniters, > > I've added some info about Pages Write Throttling and WAL compaction to > the "Ignite Persistent Store - under the hood" document. > Please let me know if anything is not clear enough. > > Best Regards, > Ivan Rakov > > > On 26.03.2018 0:18, Dmitry Pavlov wrote: > >> Hi Igniters, >> >> I've done several updates in section >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >> thehood-Regionandsegmentstructure >> >> These updates releated to optimizations of LoadedPageTable in Ignite 2.5 >> (Replacement of storage algorithm corresponding to the ID in the page, and >> the location of a physical address within the segment algorithm is used >> Robin Hood with backward shift. Previous implementation was not resistant >> to the ‘page rotation’, a lot of the cells were flooded with Removed >> state, >> and new algorithm does not have ‘removed’ state of cell.) >> >> New section has now picture of segments structure, and describes locking >> optimisations in 2.5. >> >> Sincerely, >> Dmitriy Pavlov >> >> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>: >> >> Hi Denis, >>> >>> I've updated the section in relation to your questions, please see >>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>> thehood-IgniteDurableMemory >>> >>> I’ve also moved paragraph related to page offset calculation and usage in >>> files into second article (PDS) >>> >>> Sincerely, >>> Dmitriy Pavlov >>> >>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: >>> >>> Dmitriy, >>>> >>> >>> First of all, thanks for putting this low level details on paper! Current >>>> and future community members will be grateful to you. >>>> >>>> Second, let me be a reviewer of the page and suggest some improvements. >>>> Until the review is over I would encourage us to hold on with any public >>>> announcements such as blog post. >>>> >>>> I’ve reviewed and edited Ignite Durable Memory section ( >>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>> thehood-IgniteDurableMemory >>>> < >>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>> thehood-IgniteDurableMemory>). >>>> Please address the questions below and I’ll move on with the new >>>> sections >>>> afterwards. >>>> >>>> 1) "Different caches and its partitions bring more complex addressing >>>> scheme but it is still possible to map from page ID to position in >>>> particular file: pageId => file + offset in this file”. >>>> >>>> Probably this appears too early in the doc because above we just >>>> introduced page ID notion with durable memory and now we’re talking >>>> about >>>> some files? Are these files of ignite persistence? What if the >>>> persistence >>>> is disabled? Honestly, the sentence gets out of the story context. >>>> >>>> >>>> 2) "Also, the tree pages can be merged into one if < 50% of space is >>>> used.” >>>> >>>> 50% of the whole memory available? Please clarify. >>>> >>>> >>>> 3) "There is one page ID table per memory segment.” >>>> >>>> It’s the same segment as it’s described in the main doc, right? >>>> https://apacheignite.readme.io/docs/memory-architecture#sect >>>> ion-memory-segment >>>> < >>>> https://apacheignite.readme.io/docs/memory-architecture#sect >>>> ion-memory-segment >>>> >>>> 4) "within unsafe segment” >>>> >>>> If we want to mention “usafe” here we need to explain what that is. >>>> >>>> 5) "If memory amount less than whole index size, Ignite still can >>>> operate.” >>>> >>>> How do we define “index size”? Is this the page size? >>>> >>>> >>>> — >>>> Denis >>>> >>>> >>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> >>>>> >>>> wrote: >>>> >>>>> Hi Igniters, >>>>> >>>>> >>>>> I am glad to introduce two new wiki articles. These articles contain >>>>> high-level description of >>>>> >>>>> - Ignite Durable Memory ( >>>>> >>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>> Durable+Memory+-+under+the+hood >>>> >>>>> ) >>>>> >>>>> - and Ignite Persistent Store ( >>>>> >>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>> Persistent+Store+-+under+the+hood >>>> >>>>> ) >>>>> >>>>> These articles cover basic principles and internal design. Pages may be >>>>> useful for new contributors during start of working with persistent >>>>> >>>> store >>>> >>>>> code base. >>>>> >>>>> >>>>> I would like to thank all contributors who helped me with my questions. >>>>> Their answers and support made it possible to start this description. >>>>> >>>>> >>>>> I will appreciate any feedback and questions. Suggestions and edits are >>>>> welcome, of course. >>>>> >>>>> >>>>> Sincerely, >>>>> >>>>> Dmitriy Pavlov >>>>> >>>> >>>> > |
In reply to this post by Ivan Rakov
Hi Ivan,
Excellent addition to the article. Thank you very much for contribution. I hope Igniters, especially Native Persistence experts will follow your lead. Sincerely, Dmitriy Pavlov пт, 30 мар. 2018 г. в 11:38, Ivan Rakov <[hidden email]>: > Igniters, > > I've added some info about Pages Write Throttling and WAL compaction to > the "Ignite Persistent Store - under the hood" document. > Please let me know if anything is not clear enough. > > Best Regards, > Ivan Rakov > > On 26.03.2018 0:18, Dmitry Pavlov wrote: > > Hi Igniters, > > > > I've done several updates in section > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure > > > > These updates releated to optimizations of LoadedPageTable in Ignite 2.5 > > (Replacement of storage algorithm corresponding to the ID in the page, > and > > the location of a physical address within the segment algorithm is used > > Robin Hood with backward shift. Previous implementation was not resistant > > to the ‘page rotation’, a lot of the cells were flooded with Removed > state, > > and new algorithm does not have ‘removed’ state of cell.) > > > > New section has now picture of segments structure, and describes locking > > optimisations in 2.5. > > > > Sincerely, > > Dmitriy Pavlov > > > > чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>: > > > >> Hi Denis, > >> > >> I've updated the section in relation to your questions, please see > >> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory > >> > >> I’ve also moved paragraph related to page offset calculation and usage > in > >> files into second article (PDS) > >> > >> Sincerely, > >> Dmitriy Pavlov > >> > >> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: > >> > >>> Dmitriy, > >> > >>> First of all, thanks for putting this low level details on paper! > Current > >>> and future community members will be grateful to you. > >>> > >>> Second, let me be a reviewer of the page and suggest some improvements. > >>> Until the review is over I would encourage us to hold on with any > public > >>> announcements such as blog post. > >>> > >>> I’ve reviewed and edited Ignite Durable Memory section ( > >>> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory > >>> < > >>> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory > >). > >>> Please address the questions below and I’ll move on with the new > sections > >>> afterwards. > >>> > >>> 1) "Different caches and its partitions bring more complex addressing > >>> scheme but it is still possible to map from page ID to position in > >>> particular file: pageId => file + offset in this file”. > >>> > >>> Probably this appears too early in the doc because above we just > >>> introduced page ID notion with durable memory and now we’re talking > about > >>> some files? Are these files of ignite persistence? What if the > persistence > >>> is disabled? Honestly, the sentence gets out of the story context. > >>> > >>> > >>> 2) "Also, the tree pages can be merged into one if < 50% of space is > >>> used.” > >>> > >>> 50% of the whole memory available? Please clarify. > >>> > >>> > >>> 3) "There is one page ID table per memory segment.” > >>> > >>> It’s the same segment as it’s described in the main doc, right? > >>> > https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment > >>> < > >>> > https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment > >>> > >>> 4) "within unsafe segment” > >>> > >>> If we want to mention “usafe” here we need to explain what that is. > >>> > >>> 5) "If memory amount less than whole index size, Ignite still can > >>> operate.” > >>> > >>> How do we define “index size”? Is this the page size? > >>> > >>> > >>> — > >>> Denis > >>> > >>> > >>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> > >>> wrote: > >>>> Hi Igniters, > >>>> > >>>> > >>>> I am glad to introduce two new wiki articles. These articles contain > >>>> high-level description of > >>>> > >>>> - Ignite Durable Memory ( > >>>> > >>> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood > >>>> ) > >>>> > >>>> - and Ignite Persistent Store ( > >>>> > >>> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood > >>>> ) > >>>> > >>>> These articles cover basic principles and internal design. Pages may > be > >>>> useful for new contributors during start of working with persistent > >>> store > >>>> code base. > >>>> > >>>> > >>>> I would like to thank all contributors who helped me with my > questions. > >>>> Their answers and support made it possible to start this description. > >>>> > >>>> > >>>> I will appreciate any feedback and questions. Suggestions and edits > are > >>>> welcome, of course. > >>>> > >>>> > >>>> Sincerely, > >>>> > >>>> Dmitriy Pavlov > >>> > > |
In reply to this post by dmagda
Denis, thanks for pointing this out.
I think that section on readme.io is good enough. I'd only recommend to highlight "write throttling algorithm" words with link to "Pages Write Throttling" section of "Persistence under the hood". What's already described on readme.io is enough for "Durable Memory Tuning" page, but if advanced user wants to know details about throttling, he'll be navigated to page with internals. Best Regards, Ivan Rakov On 30.03.2018 19:48, Denis Magda wrote: > Hi Ivan, > > Should we update the write-throttling section on readme.io? It makes sense > to do if any recommendations have changed: > https://apacheignite.readme.io/docs/durable-memory-tuning#section-pages-writes-throttling > > The compaction technique will be introduced and described on readme: > https://issues.apache.org/jira/browse/IGNITE-8090 > > -- > Denis > > On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <[hidden email]> wrote: > >> Igniters, >> >> I've added some info about Pages Write Throttling and WAL compaction to >> the "Ignite Persistent Store - under the hood" document. >> Please let me know if anything is not clear enough. >> >> Best Regards, >> Ivan Rakov >> >> >> On 26.03.2018 0:18, Dmitry Pavlov wrote: >> >>> Hi Igniters, >>> >>> I've done several updates in section >>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>> thehood-Regionandsegmentstructure >>> >>> These updates releated to optimizations of LoadedPageTable in Ignite 2.5 >>> (Replacement of storage algorithm corresponding to the ID in the page, and >>> the location of a physical address within the segment algorithm is used >>> Robin Hood with backward shift. Previous implementation was not resistant >>> to the ‘page rotation’, a lot of the cells were flooded with Removed >>> state, >>> and new algorithm does not have ‘removed’ state of cell.) >>> >>> New section has now picture of segments structure, and describes locking >>> optimisations in 2.5. >>> >>> Sincerely, >>> Dmitriy Pavlov >>> >>> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>: >>> >>> Hi Denis, >>>> I've updated the section in relation to your questions, please see >>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>> thehood-IgniteDurableMemory >>>> >>>> I’ve also moved paragraph related to page offset calculation and usage in >>>> files into second article (PDS) >>>> >>>> Sincerely, >>>> Dmitriy Pavlov >>>> >>>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: >>>> >>>> Dmitriy, >>>> First of all, thanks for putting this low level details on paper! Current >>>>> and future community members will be grateful to you. >>>>> >>>>> Second, let me be a reviewer of the page and suggest some improvements. >>>>> Until the review is over I would encourage us to hold on with any public >>>>> announcements such as blog post. >>>>> >>>>> I’ve reviewed and edited Ignite Durable Memory section ( >>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>>> thehood-IgniteDurableMemory >>>>> < >>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>>> thehood-IgniteDurableMemory>). >>>>> Please address the questions below and I’ll move on with the new >>>>> sections >>>>> afterwards. >>>>> >>>>> 1) "Different caches and its partitions bring more complex addressing >>>>> scheme but it is still possible to map from page ID to position in >>>>> particular file: pageId => file + offset in this file”. >>>>> >>>>> Probably this appears too early in the doc because above we just >>>>> introduced page ID notion with durable memory and now we’re talking >>>>> about >>>>> some files? Are these files of ignite persistence? What if the >>>>> persistence >>>>> is disabled? Honestly, the sentence gets out of the story context. >>>>> >>>>> >>>>> 2) "Also, the tree pages can be merged into one if < 50% of space is >>>>> used.” >>>>> >>>>> 50% of the whole memory available? Please clarify. >>>>> >>>>> >>>>> 3) "There is one page ID table per memory segment.” >>>>> >>>>> It’s the same segment as it’s described in the main doc, right? >>>>> https://apacheignite.readme.io/docs/memory-architecture#sect >>>>> ion-memory-segment >>>>> < >>>>> https://apacheignite.readme.io/docs/memory-architecture#sect >>>>> ion-memory-segment >>>>> >>>>> 4) "within unsafe segment” >>>>> >>>>> If we want to mention “usafe” here we need to explain what that is. >>>>> >>>>> 5) "If memory amount less than whole index size, Ignite still can >>>>> operate.” >>>>> >>>>> How do we define “index size”? Is this the page size? >>>>> >>>>> >>>>> — >>>>> Denis >>>>> >>>>> >>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> >>>>> wrote: >>>>> >>>>>> Hi Igniters, >>>>>> >>>>>> >>>>>> I am glad to introduce two new wiki articles. These articles contain >>>>>> high-level description of >>>>>> >>>>>> - Ignite Durable Memory ( >>>>>> >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>> Durable+Memory+-+under+the+hood >>>>> >>>>>> ) >>>>>> >>>>>> - and Ignite Persistent Store ( >>>>>> >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>> Persistent+Store+-+under+the+hood >>>>> >>>>>> ) >>>>>> >>>>>> These articles cover basic principles and internal design. Pages may be >>>>>> useful for new contributors during start of working with persistent >>>>>> >>>>> store >>>>> >>>>>> code base. >>>>>> >>>>>> >>>>>> I would like to thank all contributors who helped me with my questions. >>>>>> Their answers and support made it possible to start this description. >>>>>> >>>>>> >>>>>> I will appreciate any feedback and questions. Suggestions and edits are >>>>>> welcome, of course. >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitriy Pavlov >>>>>> >>>>> |
Ivan,
Sure, not a problem. Did more, created a hard-not-to-notice block titled Pages Write Throttling in a Nutshell. Thanks! Denis On Fri, Mar 30, 2018 at 10:00 AM, Ivan Rakov <[hidden email]> wrote: > Denis, thanks for pointing this out. > > I think that section on readme.io is good enough. > I'd only recommend to highlight "write throttling algorithm" words with > link to "Pages Write Throttling" section of "Persistence under the hood". > What's already described on readme.io is enough for "Durable Memory > Tuning" page, but if advanced user wants to know details about throttling, > he'll be navigated to page with internals. > > Best Regards, > Ivan Rakov > > > On 30.03.2018 19:48, Denis Magda wrote: > >> Hi Ivan, >> >> Should we update the write-throttling section on readme.io? It makes >> sense >> to do if any recommendations have changed: >> https://apacheignite.readme.io/docs/durable-memory-tuning#se >> ction-pages-writes-throttling >> >> The compaction technique will be introduced and described on readme: >> https://issues.apache.org/jira/browse/IGNITE-8090 >> >> -- >> Denis >> >> On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <[hidden email]> >> wrote: >> >> Igniters, >>> >>> I've added some info about Pages Write Throttling and WAL compaction to >>> the "Ignite Persistent Store - under the hood" document. >>> Please let me know if anything is not clear enough. >>> >>> Best Regards, >>> Ivan Rakov >>> >>> >>> On 26.03.2018 0:18, Dmitry Pavlov wrote: >>> >>> Hi Igniters, >>>> >>>> I've done several updates in section >>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>> thehood-Regionandsegmentstructure >>>> >>>> These updates releated to optimizations of LoadedPageTable in Ignite 2.5 >>>> (Replacement of storage algorithm corresponding to the ID in the page, >>>> and >>>> the location of a physical address within the segment algorithm is used >>>> Robin Hood with backward shift. Previous implementation was not >>>> resistant >>>> to the ‘page rotation’, a lot of the cells were flooded with Removed >>>> state, >>>> and new algorithm does not have ‘removed’ state of cell.) >>>> >>>> New section has now picture of segments structure, and describes locking >>>> optimisations in 2.5. >>>> >>>> Sincerely, >>>> Dmitriy Pavlov >>>> >>>> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <[hidden email]>: >>>> >>>> Hi Denis, >>>> >>>>> I've updated the section in relation to your questions, please see >>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>>> thehood-IgniteDurableMemory >>>>> >>>>> I’ve also moved paragraph related to page offset calculation and usage >>>>> in >>>>> files into second article (PDS) >>>>> >>>>> Sincerely, >>>>> Dmitriy Pavlov >>>>> >>>>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <[hidden email]>: >>>>> >>>>> Dmitriy, >>>>> First of all, thanks for putting this low level details on paper! >>>>> Current >>>>> >>>>>> and future community members will be grateful to you. >>>>>> >>>>>> Second, let me be a reviewer of the page and suggest some >>>>>> improvements. >>>>>> Until the review is over I would encourage us to hold on with any >>>>>> public >>>>>> announcements such as blog post. >>>>>> >>>>>> I’ve reviewed and edited Ignite Durable Memory section ( >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>>>> thehood-IgniteDurableMemory >>>>>> < >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under >>>>>> thehood-IgniteDurableMemory>). >>>>>> Please address the questions below and I’ll move on with the new >>>>>> sections >>>>>> afterwards. >>>>>> >>>>>> 1) "Different caches and its partitions bring more complex addressing >>>>>> scheme but it is still possible to map from page ID to position in >>>>>> particular file: pageId => file + offset in this file”. >>>>>> >>>>>> Probably this appears too early in the doc because above we just >>>>>> introduced page ID notion with durable memory and now we’re talking >>>>>> about >>>>>> some files? Are these files of ignite persistence? What if the >>>>>> persistence >>>>>> is disabled? Honestly, the sentence gets out of the story context. >>>>>> >>>>>> >>>>>> 2) "Also, the tree pages can be merged into one if < 50% of space is >>>>>> used.” >>>>>> >>>>>> 50% of the whole memory available? Please clarify. >>>>>> >>>>>> >>>>>> 3) "There is one page ID table per memory segment.” >>>>>> >>>>>> It’s the same segment as it’s described in the main doc, right? >>>>>> https://apacheignite.readme.io/docs/memory-architecture#sect >>>>>> ion-memory-segment >>>>>> < >>>>>> https://apacheignite.readme.io/docs/memory-architecture#sect >>>>>> ion-memory-segment >>>>>> >>>>>> 4) "within unsafe segment” >>>>>> >>>>>> If we want to mention “usafe” here we need to explain what that is. >>>>>> >>>>>> 5) "If memory amount less than whole index size, Ignite still can >>>>>> operate.” >>>>>> >>>>>> How do we define “index size”? Is this the page size? >>>>>> >>>>>> >>>>>> — >>>>>> Denis >>>>>> >>>>>> >>>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <[hidden email]> >>>>>> wrote: >>>>>> >>>>>> Hi Igniters, >>>>>>> >>>>>>> >>>>>>> I am glad to introduce two new wiki articles. These articles contain >>>>>>> high-level description of >>>>>>> >>>>>>> - Ignite Durable Memory ( >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>>>> >>>>>> Durable+Memory+-+under+the+hood >>>>>> >>>>>> ) >>>>>>> >>>>>>> - and Ignite Persistent Store ( >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+ >>>>>>> >>>>>> Persistent+Store+-+under+the+hood >>>>>> >>>>>> ) >>>>>>> >>>>>>> These articles cover basic principles and internal design. Pages may >>>>>>> be >>>>>>> useful for new contributors during start of working with persistent >>>>>>> >>>>>>> store >>>>>> >>>>>> code base. >>>>>>> >>>>>>> >>>>>>> I would like to thank all contributors who helped me with my >>>>>>> questions. >>>>>>> Their answers and support made it possible to start this description. >>>>>>> >>>>>>> >>>>>>> I will appreciate any feedback and questions. Suggestions and edits >>>>>>> are >>>>>>> welcome, of course. >>>>>>> >>>>>>> >>>>>>> Sincerely, >>>>>>> >>>>>>> Dmitriy Pavlov >>>>>>> >>>>>>> >>>>>> > |
Free forum by Nabble | Edit this page |