Igniters,
I want to be able to add another storage for Ignite like column-oriented storage. This requires some changes and defining the abstraction layer. I don't have a definite plan yet for how this could be done? Who wants to take part in this activity? I need the approval of committers for it. https://cwiki.apache.org/confluence/display/IGNITE/IEP-57%3A+API+to+implement+different+storage+engines%2C+modularization |
Hello!
My opinion is that we should have a second storage option first, add an abstraction layer later. It's not possible to make a good abstraction layer out of 1 implementation anymore than it's possible to definitely draw a straight line through one point. WRT columnar storage itself, it is an interesting idea. Regards, -- Ilya Kasnacheev пт, 18 сент. 2020 г. в 13:48, Maksim Stepachev <[hidden email]>: > Igniters, > > I want to be able to add another storage for Ignite like column-oriented > storage. This requires some changes and defining the abstraction layer. > I don't have a definite plan yet for how this could be done? > Who wants to take part in this activity? > I need the approval of committers for it. > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-57%3A+API+to+implement+different+storage+engines%2C+modularization > |
Hi Maksim,
I think your proposal is very good and useful not only for introducing new types of storage engine but also it's a good trigger to rework the existing storage engine to decouple its components of each other. I have a key question that we should answer. Where we should place our abstraction point? If we talk about another kind of B-tree we can relatively easily re-use our page memory and persisting mechanics, so the problem, in this case, is to implement the right data layout and operations over page memory. Other things (persisting, recovery) will be applied almost automatically. If we talk about something completely different like LSM we need to solve different problems to achieve acceptable performance a) Split entries by columns and store them continuously to each other as independent chunks. We need also to handle them in batch mode to achieve vectorization power. b) LSM is an append-only engine while our page memory is implemented for random-access. We need to support different data layout in memory and on disk. This also requires a different persisting mechanism rather than we have. c) Even if we do all these things we will face with new problems related to merging and compacting our data chunks [1] This will require a lot of time to bring it to production-ready form with acceptable performance, while resources are limited and there are a lot of OLAP databases that solve these problems more efficiently. I think we should focus on OLTP cases and row-oriented data layout and do things that match with our page memory and persisting mechanics. [1] https://www.scylladb.com/2018/01/31/compaction-series-leveled-compaction/ пт, 18 сент. 2020 г. в 13:56, Ilya Kasnacheev <[hidden email]>: > Hello! > > My opinion is that we should have a second storage option first, add an > abstraction layer later. > > It's not possible to make a good abstraction layer out of 1 implementation > anymore than it's possible to definitely draw a straight line through one > point. > > WRT columnar storage itself, it is an interesting idea. > > Regards, > -- > Ilya Kasnacheev > > > пт, 18 сент. 2020 г. в 13:48, Maksim Stepachev <[hidden email] > >: > > > Igniters, > > > > I want to be able to add another storage for Ignite like column-oriented > > storage. This requires some changes and defining the abstraction layer. > > I don't have a definite plan yet for how this could be done? > > Who wants to take part in this activity? > > I need the approval of committers for it. > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-57%3A+API+to+implement+different+storage+engines%2C+modularization > > > |
Free forum by Nabble | Edit this page |